A former cabinet minister was deeply pessimistic at the beginning of last week about the Labour party’s ability to mercy-kill injured leaders. The polls were showing no improvement for Gor-don Brown. The prime minister’s recovery strategy wasn’t biting. Yet there was no rebellion.

“It’s not in our DNA as a party,” he groaned. “At times like this I question whether Tony Blair was right in saying we are the natural party of government – we are just not ruthless enough.”

To get rid of Margaret Thatcher, the Conservatives could call on a stalking-horse candidate in Anthony Meyer and a proper challenger in Michael Heseltine. If you think the Conservatives are a ruthless exception then look at the Liberal Democrats, who in recent times have discarded leader after leader. It is no accident that Nick Clegg, the incumbent, is boldly gambling on tax cuts, after a decade in which his party consistently supported tax rises. One confidant says Clegg “can’t play it safe . . . his predecessors got the chop for doing too little”. Labour even failed to get rid of the hopeless Michael Foot, so why should it do any better now?

Two principles clash. First the Labour party is reluctant to fire its leader: until Blair’s difficulties it hadn’t got rid of its chieftain since George Lansbury, who preached peace in the era of the dictators before the war. But the logic of modern politics is inescapable. Parties and governments won’t go down to certain defeat without contemplating a change at the top first.

Loyalty at first won. David Miliband, the foreign secretary and favourite of the Blairites, let it be known in July that if there was a leadership election he would be available. Nobody rallied to his standard. Charles Clarke, the former home secretary, told Brown his time was up. He was greeted with deafening silence. The trade union leaders rumbled and grumbled last week but stayed their hand. Derek Simpson, the leader of Unite, said of Miliband: “We might as well elect Cameron,” adding a few choice expletives about his ability and parental legitimacy.

But more terrible polls confirmed what everyone knew: Labour is heading for elec-toral meltdown. The ultra-Blairite former cabinet ministers Alan Milburn and Stephen Byers, the so-called “usual suspects”, have lost traction and couldn’t lead. So now it is the turn of the unusual suspects to act. Siobhain McDonagh, the government whip sacked for demanding nomination papers for a leadership election, is no friend to Brown but no constant rebel either. She and her sister Margaret, the party’s first female general secretary and Blair’s hatchet woman, have hitherto avoided political games.

Significantly Joan Ryan, the MP who hit the airwaves yesterday to call for change, nurses a marginal north London seat. Labour’s southern discomfort is palpable; many of these MPs fear that the Brown recovery strategy ignores the southeast. A dozen MPs, including Patricia Hewitt, have signed an article in Progress magazine condemning the “malaise” in Downing Street.

Brown’s supporters have tried to dismiss the rebels as isolated no-hopers, but their numbers may grow. Now No 10 sniffs a plot by John Reid, one of Brown’s oldest and most ferocious political enemies. Reid, a former jack of all cabinet trades, has remained silent since he resigned from the Home Office at the end of Blair’s tenure. McDonagh was his parliamentary private secretary. I have a suspicion that if Brown does finally fall, Reid may be there at the crime scene, but his modus operandi is to deliver a Glasgow kiss, not a stab in the back.

The fact is, the backbench rebels, Blair-ite or not, want a reluctant cabinet to act. Their timing may be poor and their coordination haphazard but their purpose is clear. Whether or not they get the fabled 70 names needed for an outright challenge, they hope their agitation will snowball and put pressure on Brown to go.

Brown’s supporters plead for time and gain a hearing from the faint of heart. Give him until next week’s party conference or the Glenrothes by-election (as yet unscheduled) or the new year to sort things out. Anything to put off the fateful day of decision. The Brownites desperately flourish an article in the Fabian Review by Peter Kellner, the YouGov pollster and Labour sympathiser, to bolster their case that the government can still win the election. They can hardly call for loyalty with a straight face given their long coup against Blair, though that won’t stop Ed Balls, the cheerfully shameless schools secretary, from trying.

Kellner argues the Tory by-election defeats of the early 1990s under Major were far greater than under Brown. The percentage loss of Tory votes at Dudley West in 1994, for instance, was far greater than Labour’s loss in Crewe and Nantwich and in Glasgow East. But in 1994 the turn-out at Dudley was very low. Most worrying for Labour today, the turnouts at Crewe and Glasgow East almost reached general election levels. The actual numbers of votes, as well as the percentage take, of the Tories and the Scottish National party went up massively. The electorate consciously sets out to punish the prime minister.

Labour MPs see the polls in marginal seats giving the Conservatives big leads; they read that the voters think Brown is lurching to the left whereas they always regarded Blair as right in the centre. Their pollsters tell them the electorate is no longer listening to what Brown has to say. So when someone like Kellner tells them David Cameron’s victory is not inevitable, the Conservative party is not trusted and few people know what it really stands for, Brown’s enemies within his party conclude: “Thanks for the information. We can still win, but only if we get rid of the prime minister.”

The centre left of the parliamentary Labour party and the trade unions, however, disagree. “It is not a change of leader, but a change of direction” is the mantra of Compass, the left-wing pressure group.

“Unless some political leader comes out with a meaningful change of direction, why should we challenge Brown? There isn’t one on offer,” says a Compass strategist who doesn’t think it is worth pulling chestnuts out of the fire for Miliband or the right. That seems to be the view of middle-of-the-road cabinet ministers such as Jack Straw too.

Opponents of a change at the top are also armed with statistics that show the voters have no enthusiasm for any particular putative Brown replacement.

However, the prime minister’s difficulty is the left’s opportunity. They are already hailing the appointment of Dawn Butler to replace McDonagh as burying new Labour.

Steering a course between an increasingly restive left and openly rebellious right while the voters defect in droves is no easy game. The most cutting charge against the prime minister is advanced by one heavyweight opponent: “The government’s purpose can’t be the personal survival of the prime minister.” He believes many of the recent housing and heating measures were “gimmicks” about Brown’s recovery, not the country’s.

Before the storm the prime minister was on confident form, in no way depressed, when I saw him a few nights ago. He had lost weight and was happy to talk widely on a host of subjects. Brown is his own last true believer, contemptuous of Cameron and eager to take on the challenge of globalisation. He really believes that, like an Old Testament prophet, he can lead his people out of the desert of economic woes to the promised land of prosperity and election triumph. But without an outright victory against a formal challenger, how can he clear the air? The sniping will continue. Golgotha can’t be far away, if you will forgive my mixed biblical metaphor.

The pressure will grow on Miliband too. His enemies say he is in danger of emulating Michael Portillo in 1995, when Major called upon his chief cabinet critics to put up or shut up in an open election contest. Portillo declined and faced derision as John Redwood thereafter assumed the mantle of challenger. The presence of nonBlairites in the rebellion has changed the conversation in the run-up to the conference. They fizzle along for now, but a lit fuse can lead to an explosion.

Times Recommends

• In the real world, the public sector must pay
• Playing politics in a land of double Dutch
• Be like the plinth people: not set in stone