Few Labour MPs believe that Gordon Brown remains an electoral asset but many are nervous of replacing him because they fear that a new leader would not have time to turn around the party’s fortunes.

The Times spoke to more than 100 Labour ministers, MPs and advisers and found many expressing their bitter disappointment in public and private at Mr Brown’s premiership.

Ivan Lewis, Health Minister, called for unity and told The Times that Crewe could mark the “beginning of the end” for Labour and the party had yet to “earn” the right to a fourth term.

He said: “The by-election could either be the beginning of the end or a call to action which unites the Labour Party from top to bottom. There is no doubt that millions of people on fixed income are feeling insecure and we need to demonstrate we are the party of fairness and aspiration. We need to roll up our sleeves and show the discipline and unity which is non-negotiable if we are to earn the right to a fourth term.”

A handful of MPs and ministers — led by Graham Stringer — said that Mr Brown should be ditched now, but a majority of MPs say that a new leader would not have the mandate to serve until May 2010, forcing them to hold an election within months, which holds the potential for complete wipeout of the party. The only topic uniting the party yesterday was anger at the handling of the Crewe & Nantwich by-election, amid fears that the “Tory toffs” campaign had killed new Labour’s aspirational agenda.

One source revealed that Stephen Carter, the Prime Minister’s right-hand man, appeared weakened by the episode. He “read the Riot Act” a fortnight ago against those using class-war tactics but was overruled by “elected politicians” around Mr Brown.

Labour MPs and members of the Government made clear yesterday that Mr Brown was in deep trouble. A senior minister said: “I’m not sure that Gordon can change. I think [his exit] is a decision that we’re all going to have to sit and think about very seriously over the next few weeks.”

Linda Riordan, defending a 3,400 majority in Halifax, told The Times: “Gordon has got three or four months to get us back on track.” Another member of the Government said that people at every level of the party were “tearing their hair out — ministers see we are walking into a car crash.”

Those close to Tony Blair stayed away from public criticism of Gordon Brown yesterday, with the Left leading the charge for change. Lynne Jones, a Birmingham MP, called for “change at the top” and John McDonnell said that Mr Brown’s relaunch had been a complete disaster.

The most significant intervention from the Left came from Neal Lawson, a former adviser to Mr Brown who heads Compass, a group instrumental in getting him into No 10. He said: “This is a failed political project and . . . we need a change of direction. Can Brown do that in the next few days or weeks? Probably not.”

John Grogan, the MP for Selby, led the calls for a reshuffle and the return of Charles Clarke and Jon Cruddas.

Many MPs revealed their unhappiness but suggested that they felt there were few options remaining. Phyllis Starkey, MP for Milton Keynes South West, said: “We do need a leadership that responds more quickly to the feedback he’s getting from his members of Parliament and I think he has already learnt his lesson.”

Rob Marris, who is defending a 2,879 majority in Wolverhampton, told The Times: “It’s a disastrous result. It’s a wake-up call. He’s not as big an asset as he was nine months ago, but he remains a towering strength.”

Times Recommends

• Tree stump 'shows image of Virgin Mary'
• BBC drops Strictly judge for younger model
• Trafalgar Square 'reduced to a spectacle'