Mesopotamia has proved treacherous for many western politicians, but few have as much right to be frustrated as John McCain. McCain has supported ousting Saddam Hussein since the first Gulf war; he strongly supported the 2003 Iraq war and lent important weight in the Senate to the plans of the man who defeated him in the 2000 primaries, George W Bush.

McCain supported Bush’s war despite bitterness at the gutter tactics that Bush had used against him in the Republican primaries and long-standing friction with the defence secretary, Donald Rumsfeld, and despite having little say over how the war was conducted.

McCain learnt his lesson soon enough, as many of us did. When Rumsfeld’s lack of planning and insouciance towards order in occupied Iraq emerged in the autumn of 2003, McCain was both furious at the incompetence and mortified by the fact that he knew, as a good Republican, he’d nonetheless have to back Bush unreservedly in the 2004 election. Even when Bush slimed the Democrat candidate John Kerry, McCain’s Vietnam war buddy, McCain grinned and bore it. The photo of the old warhorse thrusting his craggy face into Bush’s bosom in the 2004 campaign will haunt him – and the rest of us – for a good while yet.

Then there was the torture issue. The revelations of prisoner abuse at Guantanamo, Bagram, Camp Cropper, Abu Ghraib and the secret CIA sites in eastern Europe shook McCain to his core. He’d been tortured for five years by the Vietnamese four decades ago. One of the things that enabled him to survive the Hanoi Hilton was the knowledge that America, the country he had fought for and loved, would never do the same to any prisoners in its own custody. And yet President Bush – the man he championed – authorised some of the very stress positions against terror suspects that to this day prevent McCain from being able to lift his arms much above his shoulders.

By 2006 McCain was ready to break completely with the Bush administration, but in keeping with his character, he could not countenance withdrawal from the battlefield. He’s an instinctual military man, a person utterly opposed to anything that could even look like American retreat, and haunted by the Vietnam experience, which he saw as a political failure, not a military one.

And unlike Rumsfeld, and other light-footprint hawks, McCain had no qualms about flooding a foreign country with as many American troops as would be required to restore order. When General David Petraeus emerged with “the surge”, a counter-insurgency plan that promised some kind of reversal of Rumsfeldism, McCain leapt at it. It’s fair to say that no American politician pioneered the surge as passionately or as presciently as McCain. Many of us felt it was too late – and that Petraeus still didn’t have sufficient troops to pull it off. McCain demurred.

And McCain was right. And lucky. I know very few experts who predicted that violence would ebb in Iraq as swiftly as it has – and some of it (such as the Sunni switch against Al-Qaeda and the Sadr militia’s quiescence) coincided with the increase in US troops, rather than being created by it.

Nonetheless, it’s unarguable that the prospects for a noncatastrophe in Iraq have vastly improved over the past 12 months. The Maliki government’s unexpected success in using the Iraqi army to suppress Sadrite militias in Basra and Sadr City has both given new life to the Baghdad government and encouraged some Sunnis to rejoin it. Then there’s oil. As the price has skyrocketed, Baghdad has just announced a no-bid contract with several leading western oil companies for development. The money should be rolling in soon enough.

So McCain is basking in success, right? Vindicated by events, he can present himself as the man who rescued the Iraq occupation and is best positioned to take it forward. Easy as pie, no? Alas for McCain, not at all.

The overwhelming response among Americans to good news from Iraq is a simple question: can we come home now? With a hefty majority still believing the war was a mistake in the first place, the “success” of the surge is less a vindication of the entire enterprise than an opportunity to get the hell out with less blowback than previously feared. Moreover, the less chaotic the situation in Iraq, the easier it is for the Democrats to persuade Americans that the relatively inexperienced Barack Obama is not that big a risk as commander-in-chief.

Withdrawal the right way, moreover, plays to Obama’s strengths, not McCain’s. McCain is a superb fighter and underdog, a man who likes his conflicts clear and his wars epic. He takes strong moral stands and sticks with them. But what is now required is a deft and subtle assessment of future military needs, a hefty dose of canny diplomacy with Iran and Syria and an ability to retain the trust of Americans that an exit is both feasible and imminent. On all these, Obama is obviously a more pragmatic choice.

You can see this in McCain’s biggest gaffe of the primary campaign. He was asked how long American troops would be in Iraq. He said he didn’t care if it were a hundred years or even a thousand years. He meant in a noncombat role, not in active warfare, but his answer revealed a core assumption: that the US will have permanent military bases in Iraq for the indefinite future, and that this is the equivalent of the long-term presence in Germany and South Korea. A pliant Arab state, fortified with US bases for the next century, and a staging post to contain Iran: these are McCain’s obvious best-case scenarios. And as the Bush administration’s plans for up to 60 permanent bases in Iraq are rejected by many Iraqi politicians, McCain’s stance begins, once again, to morph into Bush’s.

For most Americans, this is not a good thing. They have no desire to keep young Americans policing the Sunni-Shi’ite fault line halfway across the globe indefinitely; most want the massive resources now being drained by Iraq to be directed homeward. And there’s enough distrust of politicians who backed this war in the first place to be suspicious of anyone who did so and who is still eager to keep troops there indefinitely.

It’s hard to see how McCain escapes this trap. If Iraq gets worse, the domestic desire to leave will grow. If Iraq improves, the domestic desire to leave will grow. Either way, McCain’s posture – stay in until the job is done and preferably, in smaller numbers, for ever – is too close to Bush’s to avoid electoral danger. It is a cruel paradox: the more he is proved right about the immediate past, the less he seems suited to run the immediate future.

McCain’s time was eight years ago – and who knows how the world would now look if he had defeated Bush. Maybe an unforeseen crisis – perhaps with Iran – will change the terrain sufficiently for this underdog to come back one more time. But the odds are against him. And in so many ways, it isn’t his fault.

Times Recommends

• A dose of defeat is just the medicine
• Protecting ourselves from Eurocreep
• After the summer of LUV, it'll still hurt