We hadn’t got far in a Today programme discussion of my new book Time to Emigrate? on Friday before the anathema fell. It came from my opponent, the modishly left-wing historian Tristram Hunt.

Slurs about racism I expected. Instead I was accused of favouring eugenics, a more original interpretation of my thesis, for which there is no evidence in the book. You do not expect much from a telly don whose written work has drawn strong criticism for its callowness, but hinting that you are a neo-Nazi for raising the issue of excessive immigration is pushing it.

The previous day the Office for National Statistics (ONS) had announced some startling new figures: Britain was taking in 1,500 immigrants a day, while 1,000 Brits left. Which rather confirmed the central premise of my book: that more people were moving out as well as in, and that a growing number of emigrants — by no means necessarily racists — were quitting because of the numbers coming in.

Earlier in the week Mervyn King, the governor of the Bank of England, had complained to a committee of MPs that it was hard to manage the economy when nobody knew how many people were in the country.

Unmoved by any of this, Hunt denied there was a problem, real or potential. In one sense he was right: for the well-born, expensively educated liberal elite he represents, there isn’t. I doubt that the Hunt dynasty (he is the son of Lord Hunt of Chesterton) will be inconvenienced too much by immigration and its social, economic and educational consequences. Less privileged folk of his generation, for whose fears about the future he clearly has a patrician contempt, will pay a heavy price if our unprecedented experiment of mass immigration goes wrong.

Immigration alone, of course, is not the only source of their problems, and there is a danger of immigrants becoming the whipping boys for every grievance. The trouble is that random population growth impacts directly on everything feeding rumbling middle-class discontent: rising taxes, rising mortgages, failing schools, the overstretched National Health Service, crime and insecurity of every kind. I do not anticipate riots or demonstrations, but a mood of semi-suppressed nastiness could gradually develop.

Think of it: 7m more people in 25 years, according to the ONS. This is the equivalent of seven more Birminghams — not a pretty thought — or another London if you prefer. All this in the most crowded country in Europe.

My book takes the form of a letter to a (fictional) 34-year-old son and his wife on average wages who, stressed out by mortgage, school and security problems, are contemplating emigration. It is for their generation, not mine, that the prospects are shaky.

In retrospect it is extraordinary how easy we had it. In 1970 we bought a Victorian house in west London of some 3,500 sq ft for £16,000, with a mortgage based on 2 times our (smallish) income. Last week a building society began offering loans of five times income. Meanwhile, as space per person shrivels, parents helping out with the deposit stare in disbelief at the few square feet that their thousands of pounds will stretch to. For those without big daddies with big money the big squeeze has begun.

Parents can be equally appalled by some of the urban neighbourhoods that their home-seeking offspring move to in order to raise their own families. The percentage of the children of minorities in primary schools has risen from 11% 10 years ago to more than 20% today (more in parts of London). This is natural and inevitable, but those who tell us that it is something to celebrate usually educate their children elsewhere: in London the number of those opting for private education is 13%, twice the national average.

I am not saying such schools are doomed, but many have been given an awesome task. The speed of change in such communities means that parents and teachers no longer know where they stand from one year to the next. Again, the contrast with my generation is stark. My earliest school days were spent on an orderly East End working-class estate, with a good school, no ethnic tensions and no British National party.

At that point the Hunts of the 1950s and 1960s were coming under challenge from the grammar school brigade: 40 years ago only a third of Oxbridge students had been privately educated. Now the figure is 50%; and, if you count the 160 remaining grammars alongside the independents, only some 25% of the Oxbridge intake comes from comprehensives — which comprise 90% of the state education system.

If this is where we start from, how likely is mass immigration, with the overcrowding and linguistic and security problems that it is bringing, to improve the educational chances of the offspring of middle-income natives? They could easily be held back at poorly performing state schools — only to be faced with increased competition from the clever children of ambitious, new-rich immigrants at university entrance level. And how can the newcomers be blamed?

Hunt’s response to problems of social promotion is to wave them aside. Lack of mobility? Such rot. Tell that to the lower and middling classes, or to the authors of a report from the London School of Economics showing that mobility has declined in recent decades, mainly through lack of access to high quality education.

Obviously there must be some immigration and of course it can benefit Britain — especially at higher income levels. But those who claim that it benefits everyone will have to explain how — unlike the governor of the Bank of England — they can do a profit and loss account, extra GNP against extra social costs, if nobody knows who’s here.

More thoughtful members of the liberal intelligentsia have begun adjusting their tune to the figures. Trevor Phillips, of the Commission for Racial Equality, insisted in this newspaper last month that “unless we have an honest debate about the difficulties of immigration and the real anxieties out there, tensions will increase”.

It would be fun to hear Hunt and Phillips head-to-head. Since Phillips and I are often saying similar things, to be ethnically even-handed Hunt would have to call our race relations watchdog a covert eugenicist, too.

Time to Emigrate? is published by Gibson Square Books at £8.99 George Walden