To understand all the fuss that surrounds Barack Obama, go back to the 2004 Democratic Convention where wooden John Kerry, with his grey hair and stiff salute, was “reporting for duty” as the party’s presidential nominee.

They had picked what is known as a “resumé candidate” whose record in Vietnam and the Senate would, they thought, give them a chance of beating President Bush.

Then along came the skinny figure of Obama with his jazz-cool looks, exotic name and caramel skin. Though virtually unknown – not yet even elected to the Senate – his speech was, by all accounts, one of the best anyone had heard in years. As good as Bill Clinton, they said, but fresher.

“There is not a black America and white America and Latino America and Asian America – there’s the United States of America!” he said. Tears ran down faces, the hall was transformed into a rainbow of joy, the party was filled with the audacity of his hope, and a new brand was born.

Now fast-forward to today. Obama is running for president and has a credible chance of winning. But the crowds that appear wherever he goes don’t get quite the same emotional hit as they did three years ago in Boston. These days he adopts a calmer, more discursive, tone in most of his speeches and though the applause at the end is warm, it is sometimes shorter than it was when he entered the hall.

So why does this electrifying speaker feel the need to insulate his audience from the full force of his power? Obama said recently that while “I can gin up folks pretty well”, that is not his objective right now, explaining that “I want to give them a sense of my thought process”. An aide later told The Times that “everyone knows he can do a great speech”, before adding, with a knowing look, that “so could Jesse Jackson and Al Sharpton”. The implication is that these black candidates for president never made it to the White House because they were seen as overemotional politicians who played into America’s deeply embedded stereotypes. Obama, by contrast, appeals to whites because he is “exciting”, not “excitable”.

This is more than stylistic adjustment: it symbolises the racial riddle in both his identity – his white mother was from Kansas, his black father a Kenyan immigrant – and that of a nation still coming to terms with what he calls the “original sin” of slavery and Jim Crow.

His answer to it is best illustrated by an anecdote he tells of watching a black state senator in Illinois denouncing a racist. A white colleague leant over to whisper in Obama’s ear: “Whenever I hear him, he makes me feel more white.” Obama responded: “It’s not always easy for a black politician to gauge the right tone to take – too angry? Not angry enough?” But he concluded that “the comment was instructive” because “rightly or wrongly, white guilt has largely exhausted itself in America”.

The US is, nonetheless, still a country obsessed by race. An African-American TV executive tells the story of how he “deblacked” his New Jersey home when he put it on the market by removing pictures of his children from the walls, along with any other evidence that a nonwhite family had lived there. “It was probably worth a couple of hundred thousand dollars to me,” he said last week, “but I still felt shame.”

Nor do you have to live among white liberals in Washington for long to realise how they have almost entirely disconnected from the majority black population in a city that remains segregated geographically along ethnic lines. One woman recently offered this explanation for why nannies are Filipinas, Vietnamese, Mexican or African but never Afri-can-American. “I would not employ one of them because you never know when they’ll turn around and say ‘The only reason you want me to do this is because I’m black’. We just don’t want to get into that.”

But the US media can also work itself into an extraordinary lather over any overt sign of racism, as it did two weeks ago when Don Imus, a middle-aged white radio-show host, referred to some mainly black members of a women’s college basketball team as “nappy-headed hos”. This remark prompted the Revs Jackson and Sharpton to campaign, successfully, for his sacking.

Obama, by contrast, was criticised by some black commentators for being slow to respond. When he did finally speak out, it was during a visit to South Carolina – which will be the first state in the South to hold a Democratic primary election next year, and where at least half of those taking part are expected to be black. So Obama could have chosen to score some cheap points. Instead he chose to attack the rap stars who popularised the language used by Imus. “We should admit to ourselves that this is not the first time we have heard the word ‘ho’ – why are we degrading our-selves?” he asked. He went on to tell his audience that they needed to take more responsibility by stopping their children watching too much TV and clearing their neighbourhoods of rubbish.

This is precisely the type of evenhanded behaviour that can irritate other black leaders, both political and cultural. For instance, the rap star Snoop Dogg defended his use of derogatory terms by saying that when he called someone a “ho” it was because that was what they were. He insisted that he would never use such language to describe a college athlete.

Though polls suggest that Obama is doing well among African-Americans, some still suggest that he is “not black enough” because his ancestors don’t include those who experienced the pain of slavery and segregation.

In America he appears to occupy a similar space to Tiger Woods, the golfer, who is also of mixed race and briefly became an object of ridicule by describing himself as “Ca-blinasian” – a tribute to his Caucasian, black, Dutch, Native American and Thai heritage. Obama has not made the same mistake, firmly declaring himself black after a racial awakening in Hawaii.

The Rev Al Sharpton refuses to back Obama. “It’s not about his genealogy, it’s his policies,” he said. “I want to see public policy statements on what kind of Justice Department we should have, police brutality, corporate discrimination. Why would anybody assume that I’ll endorse someone without knowing the answers to those questions?”

Obama, who says that his parentage means that he has “never had the option of restricting my loyalties on the basis of race”, undoubtedly has less to be angry about.

But younger black leaders like him also reflect a changing society which has had time to come to terms with civil rights and the social ructions thrown up by the 1960s. Though it was only in 1967 that the US Supreme Court overturned laws in 15 states barring interracial marriages, the number of such unions has soared – from 65,000 in 1970 to 422,000 in 2005.

White opinion has changed, at least on the surface. In 1958, asked if they would vote for a black candidate, 53 per cent of white voters said that they would not. By 2003 it was just 6 per cent. According to a recent poll, Obama’s smoking habit is a bigger barrier to winning votes than his race.

Obama himself says: “Whatever preconceived notions white Americans may continue to hold, the overwhelming majority of them these days are able – if given time – to look beyond race in making judgments of people.

But, he insists, he does not represent “postracial politics”. Though “things have gotten better, better isn’t good enough”. In South Carolina he visited the South Florence High School, which lies in a “corridor of shame” where black graduation rates are 20 or 30 per cent below those of whites, and teachers are paid less than elsewhere in the state. The school principal, Neal Vincent, acknowledged that “there is a problem with African-American attainment”, before adding: “It’s difficult in this state to talk about this. I’m real careful not to say the N-word or to call people coloured.” Such a tendency to miss – or avoid – the point is apparent in teachers, journalists and even black politicians. It reflects a desperately stale, one-dimensional quality to the race debate in America.

Phil Noble, a (white) political strategist in South Carolina, believes that Obama, despite – or perhaps because of – his contradictions, offers a way out of this maze. “He has the potential to change our notion of who we are. You can almost hear the cracking of the eggshell of the next generation of America, and the young chick inside looks like him.

“He can have the same effect on politics that Tiger Woods had on golf when the number of people watching rose by 90 per cent, not because he was black but because he was different.”

Obama may turn out to be everything expected of him, or nothing. But for now, at least, he really is something else.

Barack Obama CV

1961 Born in Hawaii to a black Kenyan father and white American mother

1963 Parents separate and father leaves. Family later moves to Jakarta, Indonesia, and Obama attends a mainly Muslim school, although he is a Christian. Family moves back to the US when Obama is 10.

1983 Graduates from Columbia University in Political Science

1991 Graduates from Harvard Law School and practises as a civil rights lawyer

1996 Elected to the Illinois Senate

2000 Makes unsuccessful Democratic run for the House of Representatives

2004 Hits national political radar with his Democratic convention speech about the need for unity and “the audacity of hope”. Elected to the U.S. senate with 70 per cent of the vote.

2005 Time magazine hails him as one of the world's 100 most influential people

2007 Announces formal candidacy for the U.S. 2008 presidential election

Times Recommends

• No cause for celebration in crushed Sarajevo
• Jazz funeral for the final victims of Katrina
• French fined for 'insulting' civil servants