The prime minister can’t say he wasn’t warned. “Unless Gordon Brown changes he will obliterate Labour,” prophesied Charles Clarke, the former home secretary, with brutal candour three months ago. The wipeout has duly begun.

The facts speak for themselves - third place behind Nick Clegg’s Liberal Democrats, a worse result than Tony Blair four years ago at the depths of the disillusionment over Iraq. And worst of all the Tories appear almost nice and definitely fashionable for the first time in decades. The victory of a genial but raffish Etonian toff in London was the stuff of socialist nightmares.

When Chris Patten, the europhile Conservative party chairman, lost his Bath seat in the 1992 general election, the right chanted “Tory gain, Tory gain”. In the wee small hours of Friday as the local election results tolled their knell, Clarke, a Blairite, was more restrained. But inside his head was there a voice roaring “new Labour gain, new Labour gain”?

Asked whether Brown can win the next general election Clarke replied “it’s possible but it will require changes” to his team, message and sense of purpose. John McDonnell, the voice of Labour’s left, agreed: “Without a radical change of direction, we are witnessing a Labour government slipping away.”

Yet what are the chances of the Brown leopard changing his spots? The prime minister's friends fear the worst. For there is another Scottish politician he resembles who was hailed as his party’s saviour and singularly failed to make the grade - his Liberal Democrat chum Menzies Campbell.

“Chatshow Charlie” Kennedy was loved by the voters for his easy manner and good humour but there was one group of voters he could never connect with: metropolitan intellectuals. The commentariat warmed to Campbell’s cool criticism of Blair’s alliance with George Bush. But when the former Olympic sprinter finally prised poor, alcoholic Kennedy out of his job, he lost his nerve in the Commons, flopped on air and lost authority. He was good at one thing but not at many things.

Brown, too, was beloved of worthy left-liberals for his concern for Third World debt, his intellectual hinterland and his Scottish bank manager’s sobriety - which contrasted so vividly with that warmongering, celebrity-obsessed, flibbertigibbet Blair.

The aftermath of Iraq gave Brown his chance, too. The result: at prime minister’s questions David Cameron has pulverised him and media interviews have generated all the humanity of a Dalek. On the Today programme last week he produced yet another shopping list of policies. He doesn’t do a good “sorry” either. As for celebrity-obsession, how about this homme serieux’s call the other week to Shakira, the sultry, midriff-baring Colombian singer, to chat about education.

Is Brown only good at doing one thing at once? He seems to be the Peter Principle in action: promoted a rung beyond the one where he was capable.

As I wrote last week, if you compare the prime minister’s position now with John Major, Brown is three months away from that “put up or shut up” moment when a leadership challenge emerged. Yes, Labour’s history is against it, but this is the 21st century.

Look what happened when James Callaghan was allowed to choose the timing of his departure after losing the 1979 election. He hung on long enough to destroy the chances of Denis Healey, his natural heir. Michael Foot and Neil Kinnock at least had the grace to clear off without being asked.

In the twilight of Blair’s rule it was made even more difficult to mount a challenge; under the Labour party rules a challenge can proceed only if the candidate gains the support of 20% of Labour MPs - that’s about 70 of them. The bold man or woman who jumps this hurdle must be endorsed by delegates at Labour’s annual conference. After that, trade unions, Labour MPs and constituency members would have to be balloted.

So dislodging Brown against his will would entail a bloodbath. And yet the case for keeping the prime minister is getting harder and harder to defend. Of course, a Sky viewers’ poll is highly unscientific but when asked whether Brown should quit 95% pushed the “go” button on Friday.

Labour MPs whistle in the dark about Major’s recovery in 1992 but in their hearts they recall his collapse in the local elections in 1995, two years before his 1997 general election disaster. Rather than go over the cliff with Brown, some Labour politicians may prefer to take the non-lemming path.

This isn’t the only view. Matthew Taylor, Blair’s former chief of staff, a Brown well-wisher of sorts, ventures on Radio 4 that the prime minister can recover as captain of the ship if he steers us safely to port through the economic storm. True, the American financial authorities may have turned things around across the Atlantic and even our own dozy Bank of England predicts that the worst of the credit crunch is over - so long as you aren’t remortgaging or selling your house at 20% below last year’s valuation. And housing downturns go on for years.

For Labour to revive after 11 years in power, economic recovery is a necessary but not sufficient condition, however. As Clarke says, the voters simply don’t know where this government is headed. MPs have told Brown to his face that they don’t know what he is for either. His cabinet lacks a compass.

Imagine you are a backbench Labour MP unexpectedly elected in your mid-thirties in 1997 and amazed to be re-elected twice. Now in the late forties, your job, no, your entire career, looks like history. Not much to show for the best years of your life. The next time the government closes a post office in your constituency you will be tempted to tell ministers where to get off. On issue after issue you will be tempted to rebel. A crisis of authority for the prime minister beckons.

Compass, the left-wing pressure group, announces that “new Labour is dead”. It and its media allies say the prime minister is just “Blair without the economic boom”. In their analysis, Brown’s abolition of the 10p rate of income tax at the expense of the child-less working poor did for him. The left’s solution is to placate the core voter with more redistribution through higher taxes on business and the upper income brackets. The Guardianistas want to attack faith schools, city academies and end private provision in health. But how long could such policies survive the exodus of more businesses from London to Dublin and a middle-class revolt against a Swedish-style tax burden with British-style public services?

Bemused loyalist MPs talk about putting the prime minister on probation for a few more months. So after Brown failed the leadership test over the election-that-never-was, the competence test over missing data, the relaunch test in January, the character test over his unfeeling response to our reduced economic circumstances and, finally, the electoral test of May 1, he gets the “authority” test.

The Labour party really does believe in positive discrimination - at least for Brown, a disadvantaged minority of one. Watch this space for signs of his authority oozing away after the next failed relaunch/reshuffle.

Blairites outside government have the most persuasive analysis. “We have lost the language of being the party of aspiration,” says one grandee. “We’ve got to create fairness by allowing middle-income earners to realise their own aspirations . . . otherwise a campaign based on righting the wrongs of the 10p tax will place us purely as the party of poverty.”

A team change might help. The prime minister for the moment can’t stomach a reconciliation with his enemies among the so called Blairite ‘ultras’ - Clarke, Alan Milburn, the former health secretary, and Stephen Byers, the former trade and industry secretary. These men represent a danger to him: they have a coherence beyond their numbers.

In the late 18th century the British crown’s right to choose ministers could be challenged in a moment of crisis. This was called “forcing the closet”. Likewise when faced with the executioner’s axe, King Gordon I may decide that admitting the ultras into his closet is the lesser evil.

Times Recommends

• It’s the beginning of the affair
• Afghanistan: still a price worth paying
• Heroes of the downtrodden