Is it all the fault of the middle classes? I ask because some of the language used in the debate over disadvantaged kids has a faintly accusatory tone to it (Yes, guilty). Or perhaps the term “middle-class” has been used for so long as a term of abuse that it is impossible to read it any more without hearing the echo of a sneer.

The middle classes read to their children. They teach them to count. They tell them about shapes and colours and the world. I don’t think anybody is complaining about that, yet. But they also use public services – health visitors, doctors, Sure Start children’s centres – and if you listen to public sector workers for more than a few minutes, you will hear protests about that. A mother’s likelihood of seeing a health visitor after giving birth actually decreases with her income; the people who most need help are those most likely to be left behind. Yes, the middle classes worry about their children. Or “waste our time”, as one health visitor put it to me.

Do they worry too much? The most astonishing critique of middle-class aspiration came last month from the Conservative education spokesman, David Willetts. “Middle-class parents”, he said, “invest far more effort in raising their kids than they did a generation ago. My parents didn’t spend time driving me around to tennis coaching or music lessons, and I didn’t love them any the less for that.” He didn’t mean it to sound like a criticism, but it did.

Ah, the pushy parent . . . steering her child into the best state school, praying his way into the local vicar’s favour to edge a place at the church school. And elbowing the bright, poor kid out of the way.

A report published on Monday proved that by the age of 3, children from disadvantaged families lag up to a year behind wealthier contemporaries in social and educational development. Mr Willetts was talking about the same thing last month, and is due to do so again today, in a speech to the Daycare Trust. He has been studying research by an economist at the Institute of Education, Leon Feinstein, which shows how less able but wealthy pupils overtake their bright but poor counterparts at five years old. The performance of the former increases sharply from 22 months to 40 months (the research did not go back earlier than 22 months), while that of the latter falls steeply.

Which is, broadly, why Labour set up Sure Start, and offered free nursery places to three and four-year-olds. But the very children with most to gain from these – from the reading and the painting and the sticking of shapes, the stimulation, the space and even the fruit – are the ones who use them least. Children’s centres, although set up in the rougher areas, have been hijacked by, yep, those middle classes again. The unemployed, teenage single mums don’t use them.

Tony Blair has been worrying about this too. In an article in The Economist two weeks ago headlined “What I’ve Learned”, the Prime Minister cautioned that rising prosperity among the poor “masks a tail of underachievers, the socially excluded. The rising tide does not lift their ships.” It was an honest admission that the New Deals and Sure Starts and tax credits of new Labour left some people, perhaps the second and third generation unemployed, behind.

David Aaronovitch suggested on these pages yesterday that it wasn’t just poverty that prevented the parents of disadvantaged kids from reading and talking to and otherwise encouraging them. But often it is. It is the chaos of lives in grinding difficulty, where the day-to-day business of finding enough food, some money for the electricity meter, a quiet corner even to get dressed or to escape the slap of a violent partner or relative is overwhelming. You cannot read quietly to a toddler – even if you can read well enough yourself – where there are five people living noisily in two rooms, one of them perhaps drunk, drug-addicted or mentally ill. You cannot create a calm environment around that, especially if you, the new mother, are barely an adult yourself. Yes it is about poverty – which is not to suggest that simply dishing out some cash is the answer.

We have an underclass, and everybody needs to look at it honestly. The Government has responded to Mr Blair’s “tail of underachievers” with a bold scheme to create a national network of “family nurses” who attach themselves to the most disadvantaged pregnant mothers of children and stick with them until their baby is two years old, intensively guiding their health, parenting skills and even continuing education. It has been wrongly pilloried as “foetal ASBOs”. In fact it is the best chance those babies, and their mothers, have, and in the United States it has shown extraordinary results. For it isn’t true, as David Cameron has protested, that “one of the great things about this country is that it matters more where you’re going than where you come from”. For these families, it isn’t true at all.

It is just possible that a consensus is emerging about early-years intervention, among the politicians at least. Today we shall hear Mr Willetts and Ed Balls, a mouthpiece for our next Prime Minister, Gordon Brown, outline their thoughts on the next steps for childcare policy. I would love to hear a ringing endorsement from both of them of the family nurse scheme, and promises of continued funding if the current pilots show their worth. I cannot see an alternative in sight, unless you believe that socially disadvantaged mums should be forced to have abortions, or have their babies put up for adoption at birth.

The middle classes are not to blame for the lives of those who have fallen behind them. But we would be to blame if, now that we have all the evidence we need, we fail to seize our best chance of helping the most vulnerable.

Times Recommends

• Nerdy cafés no longer maid in heaven
• Cameron will not break marriage vow
• Avatar will make the case on climate