What would David Cameron most like from the prime minister? The answer, surprisingly, is a great clunking fist to his jaw.

I kid you not. Cameron confides that his gleeful ambition is “to get Gordon to really lose it in the Commons and belt me one on the chin”.

Boys, boys! That really would be Punch and Judy politics.

Last Wednesday at prime minister’s questions, the Tory leader goaded Brown so much he almost scored his masochistic goal. The young picador repeatedly taunted and jabbed his bull-like opponent across the dispatch box. Ominously, as Tory MPs bayed for more blood, the Labour benches were largely silent.

In a bid to win our sympathy, Labour spokesmen tried to compare Cameron to a “public school bully”. It has come to a pretty pass when Great Gordon – the man, remember, who relentlessly ground down Tony Blair and turned civil servants and cabinet ministers into quivering wrecks from his lair in the Treasury – is likened by his own side to poor Tom Brown taking a beating from Flashman.

Brown’s enemies within Labour are if anything more scathing. Alluding to Brown’s most recent publication, Courage, one put it thus on Have I Got News for You: “Courage! What next? Bill Clinton on chastity, Jeffrey Archer on veracity?”

Of course, the effects of Brown’s bottling of the election and the ragged aftermath in the Commons last week can be exaggerated. He still has a commanding majority in the Commons.

Yet a price has already been paid for Brown’s indecision. The Conservative leader has earned a breathing space in which to develop his policies while George Osborne, the shadow chancellor goes to work on more tax Exocets. Cameron – or “Flashie” as perhaps we should now call him – is on a mission to mend his ties to the wider “Conservative family” too. So the Tory road ahead looks, if not trouble free, then clearer than it did. But where does Brown go from here?

Last week in a flurry of insults he was condemned for having no “vision”; indulging in “followership” for stealing Tory clothes on inheritance tax; worse still he has “lost the battle of ideas”. A Brown critic foams: “We are not just policy-light but policy-free.” Without Tony Blair to fight, Brown has been a bit lost. Judging by our front page report, his beloved enemy no longer rests in peace: but this is criticism he could do without.

The prime minister’s defenders are swift to rebut these slurs. “Read Gordon’s speech. It’s about personalisation of public services,” is the ungainly answer one senior cabinet minister gave me last week when I asked about the “vision thing”. A big idea or an expensive case of patch and mend?

A cabinet minister defends playing games with an election and stealing Tory tax plans. He even denies there is a problem: “C’mon, politics is all about opportunism.” But a reading of Machia-velli’s The Prince suggests that a successful Machiavel hides his scheming from the world.

He adds “I wouldn’t have a fight-back strategy at all. We have to get on with the business of government and show how the flaky Tories have shifted to the right.” A former spin doctor piles in: “It was Blair who was obsessed by narrative. Narrative is history. Why can’t Gordon just succeed by doing things right?”

Who could deny that nothing succeeds like success? But there is still a hole in the doughnut. Take the example of one of the prime minister’s more intriguing appointments, that of the bright and able Ed Balls to his multi-tasked department. A few weeks ago Balls bravely abolished the Qualifications and Curriculum Authority, a body guilty of presiding over grade inflation in exams.

Does that mean Balls believes in more rigorous standards – which I applaud – but not structural changes, which the Blairites concluded was the only way to deliver them? We are not told. In his department ardent reformer Lord Adonis continues to build up his city academy schools. Two steps forward for structures. But Balls gives dismal local authority bureaucrats more say in how these schools will be run: two steps back.

Brown’s own wider “family” is unsettled. The left-wing pressure group Compass warns he is backsliding and asks “What is Brownism?” Other allies are dismayed that by raising the threshold of inheritance tax Brown has sold his social democratic soul.

And who is supposed to present the Brown view when Gordon’s in the bunker? Last week on Newsnight Michael Howard and a perky Charles Kennedy were good expositors of their parties’ respective thinking. Labour could only roster Roy Hattersley preaching old Labour egalitarianism.

Brown’s inner circle is tightly drawn – a cabal of young men rewarded with power for refining his initiatives, not for challenging them. When they disagree with their master (a dangerous business at the best of times) it’s about tactics, seldom about ideas.

By contrast, Cameron has Steve Hilton, his ultra-modernising strategist, and Osborne, a moderniser reassuring to the old Thatcherite right and keen to pursue robust Tory themes, who pull him in different directions. There is a bit of tension here but the Conservative leader clearly likes it that way: it’s the grit in the oyster that creates the pearl.

Brown, like Blair, is said to have a big tent. In fact he has erected a lot of little tents. A tiny one for Tories such as John Bercow and Patrick Mercer who join his commissions. An isolated one for mighty Margaret Thatcher. A comfortable one for the younger Blairites who get good berths but no real influence over policy.

But none of them get into Gordon’s great wigwam for a real powwow.

Look at Labour’s muddle over inheritance tax. One of the triggers for the coup against Blair two summers ago was a speech by his ally Stephen Byers, a former cabinet minister. His crime was to call for the easing of the burden of the tax on the middle classes before the Tories got round to it. Back then, the Brownites saw every Blairite idea as a massive provocation. And irony of ironies, Alistair Darling – who as chancellor has just lifted the tax’s thresholds skywards – was dispatched to kill the idea and Byers. Well, who’s sorry now?

It is a further irony that Brown lifts Tory ideas but not Blairite ones.

“The public wants a centrist party that will reform the public services,” says one of Brown’s critics. “They were promised it in 1997 and we took too long to give it to them. If we vacate that ground now because it was associated with Blair we will hand it all over to the Tory party.”

I do not suggest that the prime minister takes back every sworn enemy: Mr Blair’s ‘friends’ may continue the endless vendetta. But he could at least use open-minded modernisers within his government for a sounding board. John Hutton at Enterprise has a different take on the family to Balls and the Cabinet Office minister, Ed Miliband, for instance.

Andy Burnham, the chief secretary to the Treasury, muddied the waters this weekend with his frank talk in The Daily Telegraph about there being “a moral case” for tax breaks for married couples. It was spun as yet another Labour U-turn to triumphant Tory thinking. In the present climate of fear and suspicion the Treasury will be itching to apply the thumbscrews and rack to their novice. Instead, Mr Brown should invite young Burnham round to No 10 to discuss his ideas.

And what of the so-called Primrose Hill set? Before David Miliband became foreign secretary, a gang of next generation modernisers open to Brown met round Miliband’s dinner table to celebrate 10 years of new Labour in power.

James Purnell has since got his pleasant reward as culture secretary. As a Guildford boy he is one of the few cabinet ministers to know by birthright the aspirations of southern voters. Miliband shuttles around the world in his grand job. But as a former No 10 policy chief and a Londoner he is worth a chat over a late night glass of whisky at No 10.

Open that wigwam, Gordon, and let in some air. Or else you’ll suffocate.

Times Recommends

• Police beg us for food. What hope is there?
• Moral darkness that engulfed Baby P
• More to Advent than cheap chocolate