The Labour Party is starting to look as if it is playing Cluedo with its leader. Will it be Colonel Mustard, Jack Straw, in the Downing Street lounge with a revolver? Will it be Professor Plum, David Miliband, in the library with the lead piping? Could Miss Scarlet, Harriet Harman, plunge the dagger between the Prime Minister's shoulderblades in the ballroom? Or perhaps the Rev Green, Charles Clarke, will use the rope in the kitchen to kill Mr Brown.

The Labour leader is still, in fact, alive - for now at least. But the plotters are increasing in number and confidence. Murder (committing or avoiding it) was the only thing on anyone's mind as MPs fled Westminster for their summer holidays at the weekend.

The wipe-out in Glasgow East was, for many, compounded by the Prime Minister's lacklustre performance at the National Policy Forum - his David Cameron-style “look no notes” speech gave “look no answers” to the party's predicament. The unions seem to be calling the shots. Mr Brown's strongest suit - the economy - has become his Achilles' heel. One Cabinet minister believes that a challenge against the leader has gone from “possible” to “probable” in the past few days. There is talk of a “prosecco plot” - conducted via mobile phones from Tuscan farmhouses and Spanish villas - similar to the “curry coup” through which the Brownites pushed out Tony Blair.

I spoke to a junior minister yesterday who claims to be prepared to resign in September in protest at Mr Brown's leadership and believes that nine other government members could follow suit - more if you include parliamentary private secretaries, the unpaid ministerial aides. “Somebody has to pull the trigger,” the minister said. “The risk of not changing the leader is now greater than the risk of changing him. We've got to be bold.”

Dozens of backbenchers are said to be ready to sign a letter asking the Prime Minister to stand down. For many MPs, this is no longer about political positioning, it is about personal survival. Labour has been pushed into third place behind the Lib Dems in some marginal seats.

Even Downing Street advisers now concede that “momentum” - a crucial force in politics - is going away from them. One of Mr Brown's closest Cabinet allies admitted: “If my party continues to fail to manage itself it could become impossible for Gordon to stay. Everyone has a terrible fever at the moment but no one is sure if it's the plague or flu.”

The temperature may fall over the summer. But conditions are not ideal for a quick recovery. There is little residual loyalty towards the Prime Minister among his Cabinet. “I don't think Gordon would die in a ditch for me so why should I die in a ditch for him?” one minister said.

Meanwhile, there is civil war in Downing Street, with the “old guard” class warriors and the “new guard” PR men barely on speaking terms. Civil servants - who can sink or save a politician - have not warmed to Mr Brown. “People say he's charming in private but it's completely untrue,” an official who has worked closely with him said. “He's incredibly rude. He doesn't remember names. His e-mails are brusque demands. And his defining characteristic is anger. I've seen him kicking furniture.”

Mr Brown is planning a fightback in September, with a new “economic plan”. But not all those around him are convinced it will work. Downing Street has been begging Whitehall departments to come up with policies that fit into its “fairness” theme, with little luck. “It will be free kettles for old people, that sort of thing,” one insider said. “The strategy is great but there's a great gaping hole in the middle where the policy should be.”

A rebranding exercise has been prepared, with Cabinet meetings held outside London and Mr Brown photographed in different, more domestic settings - around a kitchen table for example - but those involved know that it could all go “hideously wrong”. The reshuffle might also backfire. The Prime Minister has been wooing Alan Milburn - he saw him again last week - with a view to offering him a big job but the überBlairite would probably turn it down. Whatever happens, the party conference will be about the leadership.

At the same time - and perhaps most importantly - the Prime Minister's potential rivals on what could be called the “modernising” wing of the party seem to be coalescing around David Miliband. I am told that James Purnell made clear to the Foreign Secretary a few weeks ago that he would not stand against him in a leadership contest if one takes place this year. Andy Burnham does not think he is ready for the top job. Alan Johnson, the former postman with the populist appeal, now sees himself as a mentor rather than a rival to the younger man. These four would make a powerful dream ticket.

Those who know Mr Miliband well say that although he funked it last year when he did not challenge Mr Brown, he now has the cojones to go for the leadership. He is planning a highly political conference speech. A small team - including Peter Hyman, Mr Blair's former speechwriter, and D. J. Collins, a former Whitehall press officer with good union links - are said to be advising him on an informal basis. With BlackBerrys and e-mail lists, there is no need to install the phone lines.

Mr Straw could put his name forward as a caretaker leader if there is a contest - although there is a growing sense among the younger generation of ministers that it is their turn for a shot at David Cameron. And there would almost certainly be a candidate from the Left - Ed Balls and Jon Cruddas are positioning to fill this slot.

Labour would then have to decide which direction it wanted to go in. The contest would not just be about personality, it would also be about policy. This is no longer just about the leader, it is also about the Labour Party deciding what it is for.

Times Recommends

• We have kit and leadership. Let’s go
• Behind the scenes at the biggest deal
• Conferences — what a waste of money