What more do black Americans want? They have already been handed the 13th, 14th and 15th Amendments, the abolition of segregation, the Civil Rights Act, the Voting Rights Act, a host of affirmative action programmes and now, unless something very strange happens to the American electorate in the privacy of the voting booth, a black president. Short of introducing the death penalty for using the N-word and dancing on the grave of General Robert E.Lee, what more can white Americans do to prove that the United States has finally become the land of equal opportunity?

Strange as it sounds, many on the extreme Right are licking their lips at the prospect of an Obama victory: they see in his triumph the death of special pleading. Like most liberals they acknowledge that a black man in the White House would represent a watershed, but they emphasise not what it says about whites but what it says about blacks. If an Obama presidency proves that the American Dream is colour blind, that racism is dying a death and that the civil rights movement is reaching its logical end point, who is left to blame for the chronic underachievement of blacks except blacks themselves? Blacks earn less than 75 per cent as whites, are twice as likely to be in poverty and by the end of eighth grade their score on standardised tests is equivalent to white pupils still in fourth grade. This is a shocking indictment, but of whom or what?

If we can all agree that Obama's rise is symbolic of the great strides America has made towards racial modernity, how do we square this with the economic and social failure of blacks after more than four decades of progressive federal intervention? Must we accept the perilous assertion, which underpinned both slavery and Jim Crow segregation, that blacks have a genetic deficiency that no amount of legislation can eradicate? Roland Fryer, a young black economist at Harvard, rejects this conclusion and uses ingenious mathematical reasoning to show why.

In a nutshell, he asks us to imagine an employer on Main Street leafing through a couple of hundred CVs. The employer is a decent chap who firmly believes that blacks and whites are born equal. But he is also well read and knows that the legacy of two centuries of discrimination has left blacks with a serious social handicap so that they are, on average, less qualified than whites.

Ideally, our employer would like to test each candidate individually for his suitability, but this is not an ideal world: he is in a competitive market and interviewing is expensive. So what does he do? In a hurry, he uses the information that black skin is typically associated with lower quality and invites more whites for interview instead. Given that his fellow employers across the nation are doing precisely the same thing, blacks soon cotton on that education is not helping their cause very much and turn their backs on school, pushing grades even lower. A self-perpetuating cycle is established, condemning blacks to permanent inequality.

This is disturbing because, to use the terminology of the economist Tim Harford in a lucid essay on the subject, everyone is behaving “rationally”. Prejudice of the redneck variety - where blacks are refused jobs on principle - is self-harming. By ignoring talented blacks in favour of ropey whites, these racists will eventually take a hit to the bottom line. But “statistical racism” is rational in the same way that charging extra to young men for car insurance is rational. Insurers do not “hate” young men, but know from experience that they are inclined to take greater risks. Lumping all young men together is discriminatory in a statistical sense, but it makes economic sense because it is far too costly to test every 20-year-old to ascertain his driving skill.

To test the prevalence of statistical racism, two American economists drafted 5,000 CVs and placed archetypal black names such as Tyrone or Latoya on half and white names such as Brendan or Alison on the other half. They then divided the white CVs into high and low quality and did the same with the black CVs.

A few weeks later the offers came rolling in, and guess what? The black candidates were 50 per cent less likely to be invited to interview. But, crucially, the researchers also found that although high-quality whites were preferred to low-quality whites, the relative quality of black CVs made no difference whatsoever. It was as if employers saw three categories: high-quality whites, low-quality whites and blacks. Is it any wonder that black children fail at school given that success is often ignored by employers?

The problem of race in America is no longer primarily one of bigotry and hatred: in that sense the progressive crusade that began with Abraham Lincoln and is likely to culminate with the presidency of Barack Obama has been a triumph. But if we judge the civil rights movement in terms of its most basic objective - social and economic equality - it has been a failure. The legislative programmes signed into law by President Johnson and the progressive decisions of the Supreme Court since 1954 have come up hard against a problem that they were not designed to solve.

Criminalising statistical racism is not the solution: it has already been illegal for more than 40 years but is nigh impossible to enforce. If a single generation of blacks left school as well educated as whites, the vicious cycle of inequality could be broken. Professor Fryer, who has been appointed chief equality officer of New York City education department, is running a randomised trial in which pupils of all races are offered individual and group-based financial incentives to get better grades. He has also proposed ways of amending affirmative action programmes to increase economic efficiency.

Mr Obama must pay close attention to this precious research, for the problem of race will not be solved by any amount of political symbolism or rhetorical pyrotechnics. His likely election merely signals the next stage in the long ascent towards racial equality; the slope remains as steep as ever.

Times Recommends

• Brown is in denial. Cuts are expected
• Fight our genetic urges? Slim chance
• Beware: dogs in mangers. Barking