One morning in late May, a former Iraqi military intelligence officer working as an American double agent walked up to the al-Qaeda ruler of west Baghdad. The exchange of words, then bullets, that followed has transformed the most volatile neighbourhood of Baghdad into an unexpected haven of calm.

It may, according to US officers, be one of the most significant gunfights since the 2003 invasion, and its ripples across Baghdad are bringing local Sunni and Shia men together to fight terrorists and militia in other neighbourhoods.

The showdown went like this: “Hajji Sabah, isn’t it time you stopped already?” Abu Abed al-Obeidi, a diminutive 37-year-old with a drooping moustache, tired eyes and a ready smile, said. “You have destroyed Amariyah,” he added, referring to the neighbourhood.

“Who are you?” Sabah, the Islamist emir, sneered. “We’re al-Qaeda. I’ll kill you all and raze your homes.”

Iraq timeline: five years of conflict

From accusations that the weapons of mass destruction dossier had been sexed-up to withdrawal of British troops from Basra

Multimedia

• Poll: Was Iraq war worth the sacrifice?

• Audio slideshow: Iraq conflict explained

• Pictures: Iraq, war and aftermath

Related Internet Links

• Read the Inside Iraq blog from our team in Baghdad

Related Links

• Terror suspect arrested 'as a bride'

• Key insurgent held as Iraq security improves

• Reopening of looted museum signals a calmer Baghdad

“You can try,” Mr al-Obeidi said.

The emir reached for his pistol. He was faster than Mr al-Obeidi, but his Glock 9mm jammed. As he turned to run, Mr al-Obeidi emptied his pistol into his back. His assault on al-Qaeda had begun.

Amariyah has experienced a startling rebirth since that western-style shootout. In May its streets were filled with corpses being picked over by stray dogs. American troops ventured in rarely. When they did, they used heavily armoured vehicles, several of which were blown apart by mines.

Now the shops and cafés are open, and schoolchildren and women stroll the streets. Mr al-Obeidi’s men patrol on foot with American troops and Iraqi soldiers.

It has been a precarious journey from al-Qaeda fiefdom to what US commanders see as a possible model for the future of Iraq. The process has not been made clearer by the mystique surrounding the enigmatic man at the centre of the revolt.

After the Americans rolled into Baghdad in 2003, Mr al-Obeidi, a sniper and military intelligence major in Saddam Hussein’s army, briefly joined the Sunni resistance. Within a year he had grown disillusioned with al-Qaeda, which had taken over the movement with the aim of sparking a civil war between Iraq’s Shia majority and Sunni minority.

In an abrupt about-face, he offered his services as an intelligence agent to the Americans. “I have a basic principle to fight anybody who is hurting my fellow citizens,” he said, surrounded by his uniformed, well-armed gunmen in his large offices in Amariyah. “That’s why I co-operated in 2004 with the Americans and started to work against al-Qaeda.”

He used his skills as a secret agent and former insurgent to infiltrate extreme Islamist groups. He has also built up a network of close comrades from Saddam’s sacked officer corps and the insurgency. This spring, dismayed by the failure of the Iraqi Government and its US allies to stem the bloodshed by al-Qaeda, he decided to act directly himself.

The May gunbattle was touch and go. Of the 150 men Mr al-Obeidi had gathered to fight, all but 15 fled when the bullets and rocket-propelled grenades started to fly.

He had divided his men into two groups, each ruthlessly targeting al-Qaeda leaders. As Mr al-Obeidi shot dead Sabah, his deputy, Zayed, was gunning down his No 2, known as Omar the Slayer. Zayed died in the shootout.

Outgunned, Mr al-Obeidi and his remaining men retreated to a mosque, where his friend, Sheikh Walid alAzzawi, an imam, announced over the minaret loudspeakers what may be the first jihad against al-Qaeda.

Mr al-Obeidi had contacted the Americans before the attack and asked them not to intervene. Unusually, the US Army consented. After a night of fighting, the rebels were down to three men. Sheikh Walid called the Americans and begged for support. They arrived in force and cut down the Islamists.

The Americans had hit a goldmine in Mr al-Obeidi. With his intelligence skills and local gunmen, they suddenly found that they could identify an elusive enemy. The former insurgents knew exactly where to find the Islamists and their weapons. Within a month Mr al-Obeidi’s men had led the Americans on a series of raids that swept the Islamists from Amariyah.

The recent turning of Sunni tribes and insurgents against al-Qaeda in western Iraq and Baghdad has become known as the Sunni Awakening. Here, however, it is more of a national awakening — Mr al-Obeidi’s 600- man force includes disillusioned Shia soldiers as well as Sunni former officers who had worked secretly for the US.

The Shia population of Amariyah, driven out by Sunni extremists, has started to return. “I wish we had a dozen Abu Abeds,” Abdelrazaq Abu Muhammad, a 66-year-old Shia who was chased from Amariyah by alQaeda and returned a couple of weeks ago, said. “He is working round the clock, watching and guarding.”

A retired schoolteacher, Umm Ahmed, a Shia woman who fled a year ago, said: “Everything we have now we owe to Abu Abed. This was the first place to be saved by a hero like Abu Abed.”

Among his men there is open hero-worship, with aides tripping over one another to light his steady stream of cigarettes or to patrol the streets with him.

The new forces, known as the Firsan al-Rafidan (Knights of the Two Rivers), have, however, proved to be a blunt instrument at times. Mr alObeidi, who still wears the Sabah’s pistol on his hip, deterred al-Qaeda from using a local print shop to produce propaganda leaflets by burning it to the ground.

When he finds his men — only half of whom are paid — occasionally stealing or extorting money, he beats them himself, and has broken his knuckles dispensing rough justice. Few questions are asked either when al-Qaeda suspects die in his house raids.

“Did Abu Abed beat people? I have no doubt,” Major Barry Daniels, of the 1st Battalion, Fifth Cavalry Regiment, who works closely with Mr al-Obeidi’s men, said. “We don’t have everything figured out yet and there’s a lot of risk in this.”

Aware of the dangers of creating yet another large, armed faction in a lawless city torn apart by militia, the US Army has created a special unit, the “roughnecks”, to work alongside them and smooth the coarser edges of Mr al-Obeidi.

“Anyone who says there is no risk in this is crazy,” Major Daniels said. “But we do our best to mitigate that risk.” He said that the best way to do so was for local militia such as Mr al-Obeidi’s to be integrated into the police force. The frustration of US commanders at the refusal of the Shia-dominated Government to make peace with Sunnis is tangible.

“The best way to reconcile is to show the Sunnis that they have a future in this country,” Major Daniels said. “If they continue to demonstrate that there’s no future, then God knows what will happen.”

There is no doubt that the bold assault on Sabah — combined with an increase in US troops in the capital — has calmed the chronic violence of Baghdad.

Inspired by his heroics and by the lucrative contracts for local militia — the US military has paid out $39 million (£19 million) so far — “concerned local citizens’ groups”, as they are known, now number 77,000 men after only six months.

Every day their new leaders come to seek advice from Mr al-Obeidi on taking on not only al-Qaeda but also violent Shia militia, such as the al-Mahdi Army, which are beholden to parties in the Government. Most of all, they want to know how to enlist the support of the Americans.

Some Iraqis worry that the creation of more armed groups will lead to the rise of warlords across Baghdad. Mr al-Obeidi swats that charge aside, saying that he wants to leave the country once he has secured his men jobs in the Iraqi security forces. Amariyah has no police force, partly because of the lack of interest in Sunni areas by the Shia Government. His men want to become legitimate policemen.

Once that happens, he hopes to get out with his surviving family. Two of his brothers died in the Iran-Iraq war in the 1980s, two more were kidnapped, mutilated and murdered by gunmen dressed in Interior Ministry uniforms two years ago. A $500,000 bounty has been put on his head by al-Qaeda and Mr al-Obeidi cannot leave Amariyah, a district walled in entirely by the Americans against car bombs.

“I lost most of my family, I haven’t had any rest for a long time. I think I’m done,” he said. “I did a good thing for Iraq. I want to live a normal life now,” he said.

Times Recommends

• Rio Tinto four held as China prepares case
• Diplomats must keep heads down and trousers up
• When in Rome should one make like Berlusconi?