I met Sandy Weddell, a Baptist minister in Easterhouse, Glasgow, at the end of a slow walk through the area’s run-down tenements. The day was bone-chillingly damp and the sky seemed to blend in with the bleak housing. In one of the blocks I peered into a stairwell and saw a child had left its teddy lying on an old pillow. I smiled, then my eyes alighted on some burnt silver foil and a blackened needle and my heart sank.

Sandy greeted me outside his church, where he ran a breakfast club for children. I told him what I had seen and he shrugged. “Easterhouse,” he said with a sigh. He then told me about two boys of six and eight whom he had found standing outside his breakfast club one winter morning. It was 6.30am and he didn’t open until seven o’clock but they had been standing there for half an hour.

Inside they told their story. The eight-year-old got his younger brother up every morning and they would go to school. There was hardly any food in the house and their mother was normally crashed out, sleeping off the effects of heroin. They didn’t know their father, although men were in and out of the house. “All too typical,” Sandy said softly. Listening to him, I just couldn’t believe how we had let this level of social breakdown continue.

Last week I and others tried to stop the government doing away with the necessity to consider a child’s need for a father when women apply for fertility treatment. Its argument centred on the rights of gay and lesbian adults not to feel discriminated against. It may seem far removed from the children of Glagsow’s poor but it seemed to me just another case of how, bit by bit, we are airbrushing men and their responsibilities from society.

Too many children are failed because their fathers play no part in their lives; I wonder how long it will be before we finally admit that we need to put men back and rebuild our fractured society.

Sandy, like so many who work in small community projects in difficult areas, tells a familiar tale of fatherless households where the women struggle, particularly with older, unruly children. It is predicted that in the next few years single-parent households in Glasgow will account for one in two of all households with children: in England and Wales the number of lone parents is approaching a third of all families with dependent children.

It costs us as taxpayers £22 billion - between £500 and £800 each per year - to pay for broken homes. As more and more children are born to cohabiting parents, it is worth noting how fragile such arrangements are. Half of all cohabiting couples will break up before their child is five. (Even with high levels of divorce the rate is only one in 12 for married couples.) At the time of the break-up, the woman’s income will fall by, on average, a third.

Having set up the Centre for Social Justice, I spend a large amount of time visiting housing estates and too often I find them full of young mums and no men. The local community groups always tell the same story - young men without any sense of responsibility, no family ties and the state picking up the bills. They hang around getting into trouble, then get caught up in petty crime and gradually move into drugs and gangs.

With few fathers around, the young boys find other role models: the drug dealer or the gang leader. But it isn’t just young men who suffer: girls do too. Studies show that it is from a father that young girls learn about empathetic unconditional love. Without this, vulnerable girls who have no father are more likely to be flattered by male attention and to be drawn into early sex, which is often regretted and unprotected.

It seems, as I look around, that the system conspires to break homes, then does its best to lower the life chances of those it is meant to care for. The benefits system is set so that if you are a couple living together, married or otherwise, you will have to work three times longer than a lone parent to get above the poverty line. Small wonder, then, according to the Institute for Fiscal Studies, that we recently had 250,000 more lone parents claiming benefits than we have lone parents in England and Wales.

People quickly learn that staying together on benefits is a mug’s game, so some at first pretend to be single; but very often this becomes a reality. So first we drive the fathers away and then, when the mothers can’t cope, we stick the children into care homes which seem to prepare them for a life of crime. More than 30% of the men in our prisons are from care homes, yet only 0.6% of our children have ever been in care.

The irony is that the great drive, quite rightly, to set women free and give them equal opportunities in life has in many parts of the country also set men free. The women are working, looking after the children and looking after the house. The pressure on those women, particularly those on low incomes, is enormous. Yet men also suffer: single men have much lower health outcomes with mortality rates 250% higher than for married men and suicide rates three times higher as well.

A few weeks ago I visited a lone parent on a housing estate in Kent. It was in a middle-class town and yet it had the highest concentration of lone parents in Europe. She was 19 years old, the same age as my daughter. I was struck by how tired she looked. The estate was a soulless place with a windswept shopping area and had absolutely nothing to recommend it. Her flat was dingy, with one side of the building subsiding; inside, the television was permanently on. Her little child ran around with a dummy in her mouth and there were no books on the shelves; just children’s videos.

She confessed that she didn’t read to her child and was amazed when I said that it was important for her development. It turned out that her sister, 17, also had a child and lived alone, a couple of streets away, and her brother, 16, had gone to live with an older woman on the estate.

She and her mother, who was in her late thirties, did not get on and the mother did not help with the child because she herself had other young children by another man. All were living on benefits.

As I left I wondered: what chance did that little girl have to break free from a life of dependency, ill health and hopelessness?

Times Recommends

• Time-worn Freddie gives up Test matches
• Brighton company 'involved in cyber attack'
• The art of survival: it's all in the mind