“I AM NOT a racist, but . . .” said the man at the door earnestly, when I asked him for his vote. “But all these people are coming over here from the Commonwealth and taking our jobs.” The man was a turban-wearing Sikh and was complaining about people coming here from New Zealand.

He was not alone. “I’m not a racist, but” was one of the phrases I heard most often when I was out on the stump in the 2001 election. “I’m not a racist, but I hate black people,” was one that stuck in the memory.

The failure to control immigration seems, on the surface, to be the hottest of hot political issues. The media believes that it is, and so do all the political parties. Yesterday’s announcement of Labour’s new points system is only the latest in a long series of efforts to gain the political upper hand.

Yet this leaves us with what I call the “immigration mystery”: when the Conservative Party made immigration control a central feature of its election campaign, it was a complete flop. Commentators are left with an issue that voters care about passionately and that has been badly mishandled by the Government, but loses rather than wins votes for the Opposition party. How does that work?

If all those people who favour the “university of life” over book learnin’ were right, I should be able to answer that question. After all, I’m the son of two immigrants, embedded in an ethnic minority community and both my home and my office are in areas with very big Asian communities. I ran for Parliament in a constituency with one of the largest Hindu-speaking populations in the country and, while working for William Hague, helped to commission any number of opinion polls on immigration. I should understand.

Unfortunately, the university of my life has turned out to be a second-rate polytechnic. The ability to unravel the immigration mystery has only come in the last week or so, while reading The New East End, a brilliant new study by Geoff Dench, Kate Gavron and the late Michael Young.

The authors returned to the East End of London more than 40 years after the publication of Young’s landmark study of family and kinship in the area. They found it a very changed place. The most noticeable difference was the number of Bangaldeshi immigrants. By 2001 a third of the population of Tower Hamlets were members of the Bangladeshi community and the same group accounted for three-fifths of school enrolments. Naturally, the political impact on the indigenous population has been immense.

You should read it for yourself, really, but let me try to summarise. The introduction of the welfare state after the Second World War was viewed by the white working class of the old East End as a reward for their sacrifices in the war years. Indeed, the speeches that announced the new measures often used this as their justification (and were often made in the East End itself). The white working class believed they had finally arrived as full members of society, no longer an underclass.

Then came the new wave of immigration from Bangladesh. The immigrants were immediately admitted to the welfare state club — club membership that the white working class believed it had fought and died to earn. More than this, traditional forms of working-class family organisation and patronage were overridden by the new welfare state arrangements (rather ironically, given the traditional Bangladeshi family set-up). Instead of houses being allocated to local families with a history of living on a particular street, perhaps fixed up by Mum so that her clan would be all around her, the welfare system simply judged applicants on their need. The white working class felt resentful. It is noticeable that women with children expressed the most racial hostility.

From the author’s account, peppered with the testimony of East Enders old and new, insights into the immigration mystery abound.

The first is that while the benefits of immigration (cultural and economic) are spread, the costs (rapid change, competition for welfare resources, foreign languages in schools, the famous “food smells” frequently mentioned in the book) are concentrated. Even within the East End, immigrant groups tend to cluster together. The members of the white working class feeling the impact of immigration most strongly are therefore relatively small in number. Elsewhere in the country, there may be plenty of talk that sounds similar, but it is an abstract question for these others, unlikely to determine their voting behaviour.

More important still is the relationship of class to immigration. The middle-class City types moving into the East End do not use welfare state resources or compete for low-wage jobs. They enjoy the benefits of a cosmopolitan atmosphere and cheap local labour. They would be horrified to talk to someone who mentioned “food smells”, would be mildly embarrassed even to read about it. Politically they support a welfare state based on individual need, rather than local credentials, because this may prove cheaper and is closer to the model of the business world.

In the East End, home to The Times and the City overflow, these middle-class voters hold the balance of power; beyond the East End they are the fickle, consumer-orientated voters that parties need in order to win elections. This is particularly true for the Tories. Whatever some in the party might argue, Conservatives will never be propelled to power by working-class voters resentful about their share of welfare resources. In the East End it was the Liberals who won the votes of the angry white working class.

For a few years now I have been wondering about the answer to the immigration mystery. It turns out I’ve been driving through it on my way to the office.

daniel.finkelstein@thetimes.co.uk

Times Recommends

• It’s the beginning of the affair
• Afghanistan: still a price worth paying
• Heroes of the downtrodden