When he assumed overall command in Iraq in February 2007, General David Petraeus described the situation as “hard but not hopeless”. It is important to remember that, in that grim time, his assessment ranked as cock-eyed optimism - not least in Washington.

Two battles were raging in Iraq, with both American and Iraqi casualties horrifyingly high and shooting upwards. Lethal attacks by insurgents or al-Qaeda on coalition forces averaged 180 a day, while Iraq's Sunni and Shia communities were simultaneously tearing at each other's guts. In parts though not all of Iraq, death squads roamed at will, kidnapping, torturing and killing in an orgy of sectarian violence that sundered neighbourhoods and had families cowering in terror behind barred doors. On General Petraeus's first day touring Baghdad, 55 corpses lay decomposing in the streets, victims of sectarian killings. The national daily average of civilian deaths had topped 80.

His mission was to implement the new strategy announced by George Bush the previous month: a surge deployment of five extra army brigades and three Marine units, aimed at reducing violence enough to create space for the economy to revive and political reconciliation to begin. That mission, tough enough in itself, was all but friendless back home in Washington.

President Bush had ordered the surge in the teeth of opposition from the Pentagon's top generals and the State Department. In Congress, not only Barack Obama (more troops would worsen the violence) but Republicans such as Senator Chuck Hagel (it would be “the most dangerous foreign policy blunder in this country since Vietnam”) were loudly against, and the bipartisan Iraq Study Group had echoed public opinion in arguing for a strategy of “managed failure” to camouflage a speedy US withdrawal - with Senator Joe Biden further arguing that partition was Iraq's inevitable and even desirable fate. John McCain's backing for the surge looked like sinking his bid for the presidency.

This week General Petraeus handed over command to his stalwart deputy, General Ray Odierno, with thanks to American and the much improved Iraqi forces for turning hard but not hopeless into “hard but hopeful”, and this time was hailed for his modesty. Incontrovertibly, Iraq on his watch has pulled back from the precipice.

Al-Qaeda in Iraq is not finished, as constant suicide bomb attacks attest; but it is no longer an existential menace. Its losses since April are reported on jihadist networks to be double its casualties in the four years from 2003 to 2007 - not least because of the Sunni “Awakening” against the nihilistic brutality of al-Qaeda's methods.

Anbar, the “unwinnable” western province that was the heartland of the bloody Sunni insurgency and also of al-Qaeda in Iraq, is in consequence now so peaceable that on September 1 it became the 11th of Iraq's 18 provinces to be handed from American to Iraqi military control.

In the south, Basra has been reclaimed from Shia militia rule (despite rather than because of Britain's inadequate and in part shameful contribution), as, for now, has the militantly Shia Sadr City area of Baghdad. Countrywide, daily attacks have fallen from around 180 last year to around 25, and there has been a drop of almost 80 per cent in civilian deaths. Street markets, even the odd swimming pool, have reopened. Despite still-dysfunctional electricity and water supplies and inefficient and corrupt public administration, the economy is picking up.

The surge has ended: the additional units are out of Iraq. The gains are holding, with monthly US military fatalities dramatically down, from a peak of 126 as the surge got under way to 18 last month. They are holding because the surge involved much more than extra US troops.

Militarily, it underpinned the switch, masterminded by General Petraeus, to a counter-insurgency strategy that moved forces out of barracks into Iraqi streets with a mission to protect the Iraqi population and earn their trust. Politically, the surge sent the all-important message that the US was not, after all, going to cut its losses and run.

That altered the dynamics in Iraq. Factions that had been jostling for power ahead of America's discomfited departure realised that the US would stay around until it could in some confidence leave Iraq to manage its own destiny. The Sunni switch to alliance with US forces was the most dramatic consequence, a turnaround that General Petraeus shrewdly encouraged and financed. Political conciliation is not yet a fact but at least it is talked about.

General Petraeus, however, no more does modesty than he does cock-eyed optimism. If he says that progress is fragile and still reversible, he must be taken seriously. It would be as big an error to declare the surge a “success”, as Mr Obama has abruptly found it expedient to do, as it was to oppose it in the first place, if doing so is a prelude to cutting American troop strengths in Iraq rapidly and “moving on”. This is perilously close to being the new Washington consensus.

It is not the Iraqi consensus. As Hoshyar Zebari, the Iraqi Foreign Minister, said this week: “What we do next is critical to the viability and endurance of any hard-won gains we have made.” Big tests are imminent.

Nouri al-Maliki's Shia-dominated Government takes over paying the wages of the Sunni “Sons of Iraq” from the US next month. It could make the huge mistake of refusing to incorporate more than a fifth of these fighters into Iraq's security forces: they could return to insurgency. It is still foot-dragging on vital laws on elections and sharing oil revenues throughout Iraq.

Mr Zebari did not say so, but until Iraq's factions get serious about sharing power a relapse into violence is a real risk; and most Iraqis know, even if they resent the American presence, that it is their insurance cover. Politically as well as militarily, the US holds the ring. There is, Mr Zebari insists, no fixed timetable for US troop withdrawal: decisions must be “conditions-led on the ground” to avoid “a vacuum of instability”. Nor must there be. There are no short cuts to stabilising Iraq. And that is not what Americans want to hear.

Times Recommends

• In the real world, the public sector must pay
• Playing politics in a land of double Dutch
• Be like the plinth people: not set in stone