Political authority is like market confidence. Once it has gone, it is difficult to get back. In a way that is neither fair nor rational a slide can quickly turn into a slump.

Just as Lloyd's shares continued to tumble yesterday so the Prime Minister's reputation is going through the floor. Gordon Brown has had a personal boom and bust every bit as real as the economic one as the political market reaches the conclusion that Labour is heading for electoral defeat.

The decision by the welfare guru David Freud to resign as a government adviser and join the Conservative frontbench is a vivid demonstration of the change in the balance of power. This was nothing to do with policy differences and everything to do with political reality.

“I got on incredibly well with James Purnell, there was no problem about policy,” he told me yesterday, “but you have to assess where you can make the most impact.” You do not have to be Mr Freud's great-grandfather Sigmund to work out that he believes the Tories will soon be in government and wants to work with them on plans that will take years to implement.

Just as public figures flocked to Labour in 1997, now celebrities and business leaders are following the polls towards the Conservatives. Last week it was Carol Vorderman throwing snowballs at David Cameron, yesterday a group of TV chefs backed Tory food-labelling plans. Eric Schmidt, the head of Google, who has his finger on the zeitgeist, has joined the Conservative business council. Richard Caring, owner of The Ivy and Le Caprice, who lent Labour £2 million in the 2005 election campaign, turned up at the Tories' recent black-and-white ball with a £70,000 donation in kind. Think-tanks, keen to attract business sponsorship, are concentrating on centre-right policies - Demos, new Labour's favourite wonk-house, recently launched a project on Progressive Conservativism.

Across Whitehall civil servants are carefully pulling back from the Labour administration. I am told that it is getting harder to recruit fast-streamers for plum jobs in ministerial private offices - the bright young things don't want to become too associated with the politicians of what they assume to be an out-going regime. Ministers have a growing sense that officials are dragging their feet on policy. “It's as if they're playing for time until the next lot get in, and that creates some tensions,” an aide says. The meetings between senior civil servants and opposition frontbenchers that take place before every election are being conducted with “more seriousness” this time.

Meanwhile, politicians are thinking increasingly of life after Gordon. Cabinet ministers are jostling for position in preparation for the leadership contest that would follow a Labour defeat. Harriet Harman - who, as deputy leader, has won one party election - is said by colleagues (and rivals) to be campaigning assiduously for the top job. Her call for a “class-war law” - putting social class on the same footing as race and gender in equality legislation - was seen by other ministers as a blatant attempt to suck up to the Labour Left. Her demand for an inquiry into whether City bonuses discriminate against women was viewed as a cynical attempt to jump on to a populist bandwagon. “Harriet's positioning like mad,” one senior figure says. “She's written off the election and she's thinking about herself.”

She is not the only one. Ministers claim that Ed Balls, the Schools Secretary, is “on manoeuvres” with speaking engagements to party groups all over the country - his speech suggesting that the global recession would be the worst for 100 years was seen as an attempt to reach beyond his ministerial brief. Ed Miliband's opposition to the third runway at Heathrow was interpreted by MPs (and No 10) as an attempt to appeal to Labour's green wing. Mr Purnell's pronouncements on child poverty are scrutinised for signs of modernising zeal. There is talk of alliances between some candidates and discussions about whom kingmakers such as Lord Mandelson and Jack Straw will back.

It matters little if any of these ministers really are preparing for a leadership contest - the idea that every intervention by the possible candidates is seen in this light shows how febrile the atmosphere in Cabinet is. Indeed, with splits over everything from bank bonuses to the third runway being instantly leaked to the press, it is increasingly apparent that discipline around Mr Brown's top table has broken down. The Prime Minister recently set up a communications sub-committee of the national economic council, chaired by Lord Mandelson, in an attempt to get the message straight on the recession - but with individual ambition taking precedence over party unity it will remain difficult for ministers to speak with one voice.

There is, I gather, a new row brewing over the long-awaited Bill of Rights and Responsibilities. The Prime Minister has been called into adjudicate because the Home Office has rejected the original draft produced by the Justice Department - “it was supposed to be about reducing the political fallout of the Human Rights Act,” a source says, “but all they came up with was a load more rights. The whole thing is a complete fiasco.”

The bars and tearooms of Westminster are buzzing with ever more bizarre rumours that Mr Brown could create a government of national unity with Vince Cable in the Cabinet, or resign to take up a post as head of a new global financial regulator. I recently had the first text message of the year saying “leadership challenge after local elections”. It won't happen but the Prime Minister should worry that his own MPs are shorting his stock. “There is a massive sense of fatalism both among MPs and civil servants,” a former Cabinet minister says. “Gordon's lost all authority. How can this go on for another 16 months?”

Times Recommends

• Caesar & Cicero: Balls & Mandelson
• The anti-sex brigade are the worst
• Small act of charity won’t change the world