Last week was a horrible one for Hillary Clinton. Her husband had thrown a wrench into her campaign to become president of the United States by declaring that he’d been against the Iraq war from the beginning - a transparent fib that reminded many Democrats of the pathological lying of the 1990s.

Two Clinton campaign staffers were then caught sending out e-mails warning that Barack Obama, her main rival for the Democrat ticket, was a closet Muslim. And one of her campaign co-chairmen raised the issue of Obama’s past drug use - something Obama had dealt with candidly years ago. Clinton was forced to apologise and her aide resigned. Grassroots Democrats were appalled at the descent into nastiness. It suggested desperation in the Clinton camp.

But everything came to a head in last Thursday’s Iowa debate between the Democratic candidates. Obama was asked by the moderator how he could claim to represent change on foreign policy when he had so many former Clinton administration officials advising him. Hillary burst into desperate laughter. “I’d love to hear him answer that,” she cackled. Obama paused, then fired: “Well, Hillary, I’m looking forward to having you advise me as well.” The audience erupted. In one moment, the Alpha Female ceded authority to the Alpha Male.

The Washington media are taken aback by Obama’s surge in the polls. They dismissed him months ago, buying into the notion that a Clinton presidency was inevitable. But they can’t ignore the facts in the key states: in Iowa, Obama is slightly ahead and has the organisational edge. In New Hampshire, Clinton’s double-digit lead has suddenly evaporated. In South Carolina, black voters have begun to switch en masse to Obama. It’s still far from over - and no one should discount Hillary Clinton - but the momentum is suddenly his.

How did it happen? Some will point to a solid, disciplined campaign on Obama’s part. Others will point to memorable moments like the one where he bested Hillary in last week’s debate. But this misses the core appeal and sustaining logic of the Obama candidacy. It transcends race; it runs deeper than the vagaries of daily campaigning; it represents a generational, cultural shift, and not just a political one. In a strange way, it even transcends Obama himself.

At its best, the Obama candidacy is about ending a war - not so much the war in Iraq, which will continue into the next decade - but the civil war within America that has prevailed since Vietnam and has crippled the country at the very time the world needs it most. It is a war about war - and culture and religion and race. And in that war, Obama - and Obama alone - offers the possibility of a truce.

The divide Obama promises to overcome is still between those who fought in Vietnam and those who didn’t, and between those who fought and dissented and those who fought but never dissented at all. The schism never went away. In fact it intensified during the Bill Clinton sex scandals in the 1990s, was deepened by the rise of the religious right, was ratcheted up by the bitterly divisive hung election of 2000, and worsened by the Iraq war. It is the great, paralysing red-blue divide that still rips America apart. Americans know this battle hurts only themselves, but they cannot get past it. Obama might allow them to.

What does he offer? First and foremost: his face. It could be an effective potential rebranding of the United States. Such a rebranding is not trivial - it’s central to an effective war strategy. The war on Islamist terror, after all, is two-pronged: a function of both hard power and soft power. We have seen the potential of hard power in removing the Taliban and Saddam Hussein. We have also seen its inherent weaknesses in Iraq, and its profound limitations in winning a long war against radical Islam. The next president has to create a sophisticated and supple blend of soft and hard power to isolate the enemy, to fight where necessary, but also to create an ideological template that works to the West’s advantage over the long haul. There is simply no other candidate with the potential of Obama to do this.

Consider this hypothetical scenario. It’s November 2008. A young Pakistani Muslim is watching television and sees that this man - Barack Hussein Obama – is the new face of America. In one simple image America’s soft power has been ratcheted up exponentially. A brown-skinned man whose father was an African, who grew up in Indonesia and Hawaii, who attended a majority-Muslim school as a boy, is now the alleged enemy. If you wanted the crudest but most effective weapon against the demonisation of America that fuels Islamist ideology, Obama’s face gets close. It proves them wrong about America in ways no words can.

The other obvious advantage that Obama has is his record on the Iraq war. He is the only significant candidate to have opposed it from the start. Whoever is in office in January 2009 will be tasked with redeploying forces in and out of Iraq, engaging America’s estranged allies and damping down regional violence. Obama’s interlocutors in Iraq and the Middle East would know that he never had suspicious motives towards Iraq, has no interest in occupying it indefinitely, and foresaw more clearly than most Americans the baleful consequences of long-term occupation.

It is worth recalling the key passages of the speech Obama gave in Chicago on October 2, 2002, five months before the war: “I don’t oppose all wars. And I know that in this crowd today there is no shortage of patriots, or of patriotism. What I am opposed to is a dumb war. What I am opposed to is a rash war ... I know that even a successful war against Iraq will require a US occupation of undetermined length, at undetermined cost, with undetermined consequences. I know that an invasion of Iraq without a clear rationale and without strong international support will only fan the flames of the Middle East, and encourage the worst, rather than best, impulses of the Arab world, and strengthen the recruitment arm of Al-Qaeda. I am not opposed to all wars. I’m opposed to dumb wars.”

The man who opposed the war for the right reasons is the potential president with the most flexibility in dealing with it. Clinton is hemmed in by her past and her generation. If she pulls out too quickly, she will fall prey to the usual browbeating from the right. If she stays in Iraq too long, the antiwar base of her own party, already suspicious of her, will pounce. The debate about the war in the next four years needs to be about the practical and difficult choices ahead of us - not about the symbolism of whether it’s a second Vietnam.

A generational divide also separates Clinton and Obama with respect to domestic politics. Clinton, as a liberal, has spent years in a defensive crouch against triumphant postRonald Reagan conservatism. Her liberalism is warped by what you might call a political post-traumatic stress syndrome. Reagan spooked people on the left, especially those, like Clinton, who were interested primarily in winning power. She suspects that the majority is not with her and so some quotient of discretion, fear or plain deception is required to advance her objectives - which is why charges of plasticness and lack of authenticity still plague her candidacy.

Obama, simply by virtue of when he was born, 1961, is free of this defensiveness. “Partly because my mother, you know, was smack-dab in the middle of the baby-boom generation,” he told me. “She was only 18 when she had me. So when I think of baby boomers, I think of my mother’s generation. And you know, I was too young for the formative period of the Sixties civil rights, sexual revolution, Vietnam war. Those all sort of passed me by.”

Obama’s mother was, in fact, born only five years earlier than Hillary Clinton. He did not politically come of age during the Vietnam era, and he is simply less afraid of the right wing than Clinton is, because he has emerged on the national stage during a period of conservative decline. And so, for example, he felt much freer than Clinton to say he was prepared to meet and hold talks with hostile world leaders in his first year in office. He has proposed sweeping middle-class tax cuts and opposed drastic reforms of social security, without being tarred as a fiscally reckless liberal. He is among the first Democrats in a generation not to be afraid or ashamed of what they actually believe, which also gives them more freedom to move pragmatically to the right, if necessary.

He does not smell, as Clinton does, of political fear. And there are few areas where this Democratic fear is more intense than religion. The crude exploitation of religious zeal by George W Bush and the architect of his campaigns, Karl Rove, succeeded in deepening the culture war, to Republican advantage. Again, this played into the boomer divide between God-fearing Americans and the peacenik atheist hippies of lore.

The Democrats have responded by pretending to a public religiosity that still seems strained. Listening to Hillary Clinton detail her prayer life in public, as she did last spring, was poignant because her faith may well be genuine; but also repellent because its Methodist genuineness demands that she not profess it so tackily. But she did. The polls told her to.

Obama, in contrast, opened his soul up in public long before any focus group demanded it. His first book, Dreams from My Father, is a candid, haunting and supple piece of writing that reveals Obama as someone whose “complex fate”, to use Ralph Ellison’s term, is to be both believer and doubter.

This struggle to embrace modernity without abandoning faith falls on one of the fault lines in the modern world. It is arguably the critical fault line, the tectonic rift that is advancing the bloody borders of Islam and the increasingly sectarian boundaries of American politics. As humankind abandons the secular totalitarianisms of the last century and grapples with breakneck technological and scientific discoveries, the appeal of absolutist faith is powerful in both developing and developed countries.

From the doctrinal absolutism of Pope Benedict’s Vatican to the revival of fundamentalist Protestantism in America and Asia to the attraction for many Muslims of the most extreme and antimodern forms of Islam, the same phenomenon has spread to every culture and place. You cannot confront the complex challenges of domestic or foreign policy today unless you understand this gulf and its seriousness. You cannot lead the United States without having a foot in both the religious and secular camps.

Here again, Obama, by virtue of generation and accident, bridges this deepening divide. He was brought up in a nonreligious home and converted to Christianity as an adult. But - critically - he is not born-again. His faith - at once real and measured - lives at the centre of the American religious experience. It is a modern, intellectual Christianity.

“I didn’t have an epiphany,” he explained to me. “What I really did was to take a set of values and ideals that were first instilled in me from my mother; you know, belief in kindness and empathy and discipline, responsibility. And I found in the church a vessel or a repository for those values and a way to connect those values to a larger community and a belief in God and a belief in redemption and mercy and justice . . . I guess the point is, it continues to be both a spiritual, but also intellectual, journey for me, this issue of faith.”

The best speech Obama has ever given was in Connecticut in June 2007. In it, he described his religious conversion: “One Sunday, I put on one of the few clean jackets I had, and went over to Trinity United Church of Christ on 95th Street on the South Side of Chicago. And I heard Reverend Jeremiah A Wright deliver a sermon called The Audacity of Hope. And during the course of that sermon, he introduced me to someone named Jesus Christ. I learnt that my sins could be redeemed. I learnt that those things I was too weak to accomplish myself, he would accomplish with me if I placed my trust in him. And in time I came to see faith as more than just a comfort to the weary or a hedge against death, but rather as an active, palpable agent in the world and in my own life.

“It was because of these new-found understandings that I was finally able to walk down the aisle of Trinity one day and affirm my Christian faith. It came about as a choice and not an epiphany. The questions I had didn’t magically disappear. The sceptical bent of my mind didn’t suddenly vanish. But kneeling beneath that cross on the South Side, I felt I heard God’s spirit beckoning me. I submitted myself to His will, and dedicated myself to discovering His truth and carrying out His works.”

To be able to express this kind of religious conviction without disturbing or alienating the growing phalanx of secular voters, especially on the left, is quite an achievement. It is both an intellectual achievement, because Obama has clearly attempted to wrestle a modern Christianity from the anachronisms of its past, and an American achievement, because it was forged in the only American institution where conservative theology and the Democratic party still communicate: the black church. And this, of course, is the other element that makes Obama a potentially transformative candidate: race.

Here, Obama again finds himself in the centre of a complex fate. His appeal to whites is palpable. I have felt it myself. Earlier this autumn, I attended an Obama speech in Washington on tax policy that underwhelmed on delivery; his address was wooden, stilted, even tedious. It was only after I left the hotel that it occurred to me that I’d just been bored on tax policy by a national black leader. That I should have been struck by this was born in my own racial stereotypes, of course. But it won me over. Obama is deeply aware of how he comes across to whites.

In a revealing passage in his first book, he recounts how, in adolescence, he defused his white mother’s fears that he was drifting into delinquency. She had marched into his room and demanded to know what was going on. He flashed her “a reassuring smile and patted her hand and told her not to worry”. This, he tells us, was “usually an effective tactic”, because people were satisfied as long as you were courteous and smiled and made no sudden moves.

They were more than satisfied, they were relieved - such a pleasant surprise to find a well-mannered young black man who didn’t seem angry all the time. And so you have Obama’s campaign for white America: courteous and smiling and with no sudden moves. This may, of course, be one reason for his still-lukewarm support among many African Americans (although that is changing). It may also be because African Americans (more than many whites) simply don’t believe that a black man can win the presidency, and so are wary of wasting their vote.

And the persistence of race as a divisive, even explosive factor in American life was unmissable the week of Obama’s tax speech. While he was detailing middle-class tax breaks, thousands of activists were preparing to march in Jena, Louisiana, after a series of crude racial incidents had blown up into a polarising conflict. Jesse Jackson voiced puzzlement that Obama was not at the forefront of the march. “If I were a candidate, I’d be all over Jena,” he remarked.

Obama didn’t jump into the fray (no sudden moves), but instead later issued a measured statement on Jena that was simultaneously an endorsement of black identity politics and a distancing from it: “When I’m president,” he said, “we will no longer accept the false choice between being tough on crime and vigilant in our pursuit of justice. Dr King said, ‘It’s not either/or, it’s both/and.’

“We can have a crime policy that’s both tough and smart. If you’re convicted of a crime involving drugs, of course you should be punished. But let’s not make the punishment for crack cocaine that much more severe than the punishment for powder cocaine when the real difference between the two is the skin colour of the people using them.”

Obama’s racial journey makes this kind of both/and politics something more than a matter of political compromise. The paradox of his candidacy is that, as potentially the first African-American president in a country founded on slavery, he has taken pains to downplay the racial catharsis his candidacy implies. He knows race is important, and yet he knows that it turns destructive if it becomes the only important thing. Nor is he a postracial figure like Tiger Woods, having spent his life trying to reconnect with a black identity his childhood never gave him.

Equally, he cannot be a Jesse Jackson. His white, single mother brought him up to be someone else. In Dreams from My Father, Obama tells how he had an almost eerily nonracial childhood, and had to learn what racism is, what his own racial identity is, and even what being black in America is.

And so Obama’s relationship to the black American experience is as much learnt as intuitive. He broke up with a serious early girlfriend in part because she was white. He decided to abandon a postracial career among the upper-middle classes of the east coast in order to re-engage with the black experience of Chicago’s South Side.

It was an act of integration - personal as well as communal - that called him to the work of community organising. This restlessness with where he was, this attempt at personal integration, represents both an affirmation of identity politics and a commitment to carving a unique personal identity out of the race, geography and class he inherited. It yields an identity born of displacement, not rootedness.

And there are also times when Obama’s experience feels more like an immigrant story than a black memoir. His autobiography navigates a new and strange world of an American racial legacy that never quite defined him at his core. He therefore speaks to a complicated and mixed identity – not a simple and alienated one. This may hurt him among some African Americans, who may fail to identify with this fellow with an odd name. But there is no reason why African Americans cannot see the logic of Americanism that Obama also represents, a legacy that is ultimately theirs as well.

To be black and white, to have belonged to a nonreligious home and a Christian church, to have attended a majority-Muslim school in Indonesia and a black church in urban Chicago, to be more than one thing and sometimes not fully anything - this is an increasingly common experience for Americans, including many racial minorities. Obama expresses such a conflicted but resilient identity before he even utters a word. And this complexity may increasingly be the main thing all Americans have in common.

None of this, of course, means that Obama will be the president some are dreaming of. His record in high office is sparse; his performances on the campaign trail have been patchy; his chief rival for the nomination, Senator Clinton, has bested him often. At times, she has even managed to appear more inherently likable than the skinny, crabby and sometimes morose newcomer from Chicago.

The paradox is that Hillary makes far more sense if you believe that times are actually pretty good. If you believe America’s current crisis is not a deep one, if you think pragmatism alone will be enough to navigate a world on the verge of even more religious warfare, and that what appears dark today is an illusion fostered by the lingering trauma of the Bush presidency, then the argument for Obama is not that strong. Clinton will do.

But if you sense, as I do, that greater danger lies ahead and that our divisions and recent history have combined to make the American polity and constitutional order increasingly vulnerable, then the calculus of risk changes. Sometimes, when the world is changing rapidly, the greater risk is caution.

Close up in this election campaign, Obama is unlikely. From a distance, he is necessary. At a time when America’s estrangement from the world risks tipping into dangerous imbalance, when a country at war with lethal enemies is also increasingly at war with itself, when humankind’s spiritual yearnings veer between an excess of certainty and an inability to believe anything at all, and when sectarian and racial divides seem as intractable as ever, a man who is a bridge between these worlds may be indispensable. We may in fact have finally found that bridge to the 21st century that Bill Clinton told us about. Its name is Barack Obama.

This is an edited version of an article that first appeared in the December issue of The Atlantic magazine

Times Recommends

• Rio Tinto four held as China prepares case
• Diplomats must keep heads down and trousers up
• When in Rome should one make like Berlusconi?