OUR walk down the dusty road in Baghdad al-Jadida, a district in the southeast of the city, was slow. The American soldiers moving cautiously forward between the high walls of the houses turned repeatedly and scanned the street behind us for threats.

The men were in camouflage uniforms and full battle gear, their M16s half-lifted and ready to fire should the enemy attack as we edged towards the mosque of a militant sheikh.

Until a few weeks ago, gunfire, bombs and rockets were a daily menace to the soldiers of the 2nd Battalion, 16th Infantry Regiment. They had been propelled into the heart of this hardline Shi’ite district by President George W Bush’s “troop surge” with orders to clear and hold it in the face of an onslaught by the Mahdi Army.

Yet the first Iraqis they encountered in the 117F heat last Friday were a little girl and her parents standing at the open gate to their courtyard, who exchanged salaam alaikums (hellos) with a soldier.

When the surge in this part of Baghdad began last March, the area was controlled by Mahdi Army followers of Moqtada al-Sadr, the radical Shi’ite cleric, who ambushed and killed coalition soldiers seemingly at will. The locals complied with their demands for money or out of fear.

The second battalion soldiers who moved in were the first to base themselves in the area since 2003, the year of the invasion. They set themselves up in Base Rustamiyah, an old Iraqi army school, and implanted small companies of men in the worst neighbourhoods. By June they were engulfed in all-out war.

Mortars and rockets pounded the base and its small outposts relentlessly. Soldiers were killed and wounded by snipers and roadside bombs. “It was more war than I ever want to see again,” said Major Brent Cummings, the second-in-command.

Then the rain of mortars relented and during the past two weeks the roadside bombs, their enemy’s most potent weapon, have almost disappeared.

The patrol I joined signified a new atmosphere. We drove from the Rustamiyah base in a convoy of seven Humvees, with 28 soldiers and two interpreters, along a dirt road through a concrete factory and a squatters’ camp of Shi’ite families driven from their homes elsewhere in Iraq.

When we got out to walk, Captain Tyler Andersen, the 30-year-old platoon leader, wanted to assess the local mosque hung with a huge poster of al-Sadr and listen to what was being preached. In the event it turned out to be deserted.

Andersen knocked on a family’s gate and was welcomed in and asked whether his soldiers wanted anything to drink. The welcome was not effusive but there was no hostility.

“This is working,” Andersen said. “We should have been doing this four years ago. We got so many things wrong. In my experience, most Iraqis just want peace and quiet and want to get on with their lives.”

The battalion’s statistics tell a stark story. In June, at the height of the fighting, 80 roadside bombs were detonated against its patrols. In August it was 19. There were 32 mortar or rocket attacks in June, but only 11 last month.

The battalion’s experience is reflected in figures released last week by General David Petraeus, the top US commander in Iraq, who will report to Congress on the progress of the surge by September 15.

In an interview with The Australian newspaper, he said that religious and ethnic killings in Baghdad were down by 75% since last year and the number of weapons caches seized had doubled between January and August. Al-Qaeda in Iraq has been largely driven from the city by a combination of US forces and Sunni fighters.

While the toll being exacted across Iraq was illustrated yesterday by figures showing that the number of civilians killed rose from 1,653 in July to 1,773 in August, some measure of order appears to be returning to parts of Baghdad as a result of the surge.

The progress is attributed not only to the increase in troop numbers but also to a change in tactics.

Before it began, US forces would often pass through an area, fight and leave, allowing the insurgents to return. Once the surge was under way, Lieutenant Colonel Ralph Kauz-larich, commander of the battalion, put three small companies in tough neighbourhoods and kept them there. “The strategy is quite simply boots on the ground,” he said.

The colonel believes his men inflicted their most significant casualties at the peak of the fighting in June and July. “We killed a lot of their first rank,” he said. His men are now facing second and third-level fighters from the Mahdi Army, although that does not mean the threat is over: Mahdi leaders have resorted to hiding improvised explosive devices (IEDs) inside dead water buffaloes.

Further tactical changes during the summer included instructions to ensure that at least two members of any foot patrol are wearing see-through protective glasses rather than dark shades so that they can make eye contact with local residents.

The battalion has started arresting anyone with weapons in their home, making it harder for the Mahdi Army to find hiding places for guns, and local people are starting to help with intelligence. “They may not love us but they realise we are going to stick around and they may as well work with us,” said the colonel.

He even came to an arrangement with a senior Mahdi Army officer who was thrown into prison, offered release if he helped the Americans and is now involved in running a $30m sewerage programme.

When I was reporting from Baghdad a year ago, the situation seemed hopeless with ethnic cleansing rife, car bombs costing hundreds of lives and scores of bodies dumped on the streets or floating in the Tigris each morning. The surge seems to have brought the possibility of success.

Huge obstacles remain. Car bombings by Sunni groups may intensify in the run-up to the Petraeus report. Shi’ite death squads are still feared. Iraqi security forces will not be ready to take over for months.

But some in Baghdad are starting to benefit from tangible improvements. This week will see the symbolic reopening of Abu Nawas Street, formerly a favourite spot for nighttime socialising which was famous for the “masgouf” fish dinners cooked on open fires. That tradition was ended by car bombs. Its return will be regarded by some Iraqis as a sign that life as it was may yet be restored.

See full version of Petraeus interview here

Times Recommends

• New York has lost its edge
• Copenhagen will fail – and quite right too
• More than 17,000 new underwater species found