During the worst days of the Major Government, I had lunch with a Cabinet minister who was in despair at the lack of direction from the top. “It's like sailing a boat in really choppy waters,” he moaned. “If you've got the wind in your sails, you can make headway through the waves. If you haven't, you just get buffeted all over the place and the water starts coming in.”

Talking to Labour MPs and ministers this week, I recognised the same sense of drift and desperation. Many are close to panic that they will lose their seats. If the party does badly in today's elections - and particularly if Ken Livingstone loses in London - fury against the Prime Minister will erupt. London is the key, with seats in the assembly as important as mayoralty, since many of Labour's marginals are in the capital. Local elections in the regions will tell us less, as Labour did so poorly there when the seats were last contested in 2004.

Brownites are whispering that their old enemies will unveil a conspiracy over the weekend, but there is not, as yet, any plot to unseat him. “It's not in our make-up,” explains a senior Blairite, who would love the Labour leader to step down voluntarily. “The trouble is, the Brownites think everyone is like them.”

Another former Cabinet minister agrees: “The obvious thing would be to organise and sort it out. But that's not what we do in the Labour Party. We tolerate incompetence and failure. We elected Michael Foot, after all.”

The difference now is that Labour is in government, not opposition, with a decent majority. At the next election, things can only get worse. There are well over 100 Labour MPs who would be destined for the jobseeker's allowance if the Tories were to win power. And most of them blame Mr Brown for their predicament.

They may not always have liked what Tony Blair did, but at least they knew why he was doing it. Nobody ever complained of his weak leadership. With Mr Brown, they don't understand why he does what he does - why abolish the 10p tax band if it was intended to help the poor? - and he is notably bad at either listening to their concerns or using charm and persuasiveness to win them and the voters over. Instead, he barks at his critics, denies the facts and even makes up some of his own. Yesterday, on the Today programme, he claimed to have taken a million children out of poverty, when the actual figure is 600,000. Inflation is hitting not only food and oil these days.

MPs and ministers desperately want Mr Brown to change, but they are not hopeful. Where are the big ideas? Where is the sense of direction? Whatever happened to the collegial approach he promised?

Loyalty to the leader is unravelling, and if today's results are bad and the Crewe and Nantwich by-election at the end of the month is lost, the speed will accelerate. Backbench MPs will decide that their best hope of keeping their seats is to dissociate themselves from Mr Brown and his policies. That spells disaster for the 42-day detention vote and probably more trouble on the 10p tax rate. Already Frank Field plans to put down an early day motion on Tuesday, spelling out the changes the Treasury has to make to avoid defeat at the report stage of the Finance Bill. If Mr Brown can't even get his key legislation through, what authority does he have left?

The Whips' Office is in turmoil, as evidenced by two recent leaks of memos from what should be the most secure wing of government. Geoff Hoon, the Chief Whip, reportedly feels undermined by Nick Brown, his deputy, who is one of the Prime Minister's closest allies. Meanwhile, several MPs who signed that letter to Mr Blair calling on him to resign in September 2006 were rewarded with Whips' jobs. Other MPs are reluctant to take lessons in loyalty from them. They ask, sardonically: “If I rebel, will I get to be a Whip?”

Farther up the food chain, there are remarkably few senior ministers prepared to go round the broadcasting studios defending their leader. The only volunteers - Jack Straw, David Miliband, John Hutton, Hazel Blears - are former Blairites. Where are Mr Brown's old friends when he needs them? Ed Balls, Douglas Alexander and Ed Miliband seem to have learnt their boss's dark art of disappearing when things look bad.

So what are the prospects of Mr Brown either bolstering his leadership or giving way to someone else? Stumbling along as before, and hoping something will turn up is not a credible option. Many MPs hope that he will become miraculously more competent, more strategic, more human, more open. But men of his age don't transform themselves overnight, and his is a particularly rigid personality. If he pretends to change, as he did last summer, he won't keep up the facade for long.

A few MPs are suggesting that he should follow Mr Major - who else? - in calling a “put up or shut up” leadership election. This is pretty implausible, not least because he would not take the risk.

So the only hope for those who are convinced that Mr Brown is leading Labour to defeat is that he can be persuaded, at some point in the coming year, to stand down in the interests of the party. That might require a delegation of senior ministers and friends, but he would present it as his own choice.

Would he even consider it? Maybe not. But it is clear that Mr Brown is not remotely enjoying the job he spent the whole of his life trying to win. I bumped into him last week, and his face was grey, cast in a rictus of tension. Rather like John Major circa 1995, in fact.

sieghart@journalist.com

Times Recommends

• Strip away the figleaf and reveal naysayers
• Dave’n’George: there may be trouble ahead
• I do not accept that £64,000 a year for politicians is peanuts