When I interviewed David Cameron for the Times magazine last autumn, I asked him if he’d ever interacted with hoodies in the streets around his home. “I’ve talked to them sometimes,” he said, a little defensively, “but I can’t remember any particular conversation.” Had he ever felt intimidated? “Yes,” he said, “sometimes I walk back from the Tube at night and, you know, under the Westway is not a great part of town. You can feel intimidated, of course you can.”

Yesterday I read that the Tory leader thinks “if children are misbehaving, we should say something. Collective disapproval is a powerful tool in regulating behaviour. We don’t want to live in a walk-on-by society.” And yet I understood from our interview that walk on by is exactly what he does. He has not, I think, personally confronted anyone over their bad behaviour, because bluntly he is too frightened.

I’m not blaming the guy. I’m frightened too. I live in South Hackney, which isn’t known as the Notting Hill of East London but may become so if I plug the soubriquet often enough, and there, as in many inner-city areas, the well-off and well-behaved live alongside people who can be loud, aggressive, occasionally downright uncivilised. And that’s just the parents, never mind their kids.

It’s a rare week that I don’t witness a crime or misdemeanour, from littering to verbal abuse to vandalism to dangerous driving. Do I do anything about it? Hardly ever. Why not? The same reason Cameron doesn’t: physical fear. Rapid risk-benefit analysis always favours doing nothing.

What is to be done? I have one modest suggestion. The big change in places such as North Kensington and South Hackney is that the respectable working class have upped and gone and the people like Cameron and myself who have replaced them are reluctant to assert ourselves in the same robust way that the people who lived in our homes 30 or 50 years ago would have done.

The way a community used to enforce social norms is that respectable working-class women would threaten miscreants with stigmatisation, and respectable working-class men, the bigger ones at any rate, would threaten them with force. Social stigma is a difficult thing to recreate. But becoming more willing and able to threaten force is actually quite easy, the work of a few months in the gym and a modest adjustment to the middle-class mindset.

No doubt there are many things to be done at the national, local and neighbourhood level (although Blair has given many of them a go already) but in terms of individuals acting rather than not acting, which is what Cameron is exhorting us to do, then, unless it’s just more hot air, ultimately he, me and lots of chaps like us have to get better at fighting. Home-Office-funded sparring sessions? Tax relief on protein shakes? Subsidised dumbbells? Come on, Dave, you need a few policies.

There are three for starters.

Times Recommends

• We have kit and leadership. Let’s go
• Behind the scenes at the biggest deal
• Conferences — what a waste of money