It was meant to be a gathering of friends. However, as they say in politics: “Your enemies are on your own side. The opposition are just opponents.”

Last weekend’s Progressive Governance Summit in the five-star surrounds of the Grove hotel near Watford, Hertfordshire, brought together the world’s leading centre-left politicians from Thabo Mbeki of South Africa to Australia’s Kevin Rudd.

Senior members of the British cabinet joined the prime minister at the event, including David Miliband, the foreign secretary, Douglas Alexander, the international development secretary, and John Hutton, the enterprise secretary. Former Blairite ministers such as Alan Milburn and Peter Mandelson were also there.

Gordon Brown was all ivory-white smiles. Unusually for him, according to onlookers, he genuinely appeared to be enjoying himself.

However, in the corners of the hotel bar, which usually sees more footballers and their wives than Labour politicians, plotting was afoot. “The talk was all about how long Gordon can go on,” said one who was present. “It had been another torrid week and the question being asked in private was: will he survive until the election?”

One former minister put forward an argument that would have been unthinkable 12 months ago. “It is valid to ask whether we are at the situation the Tories were in the autumn of 1990, when they pondered the coup d’état against Margaret Thatcher,” he said.

After the conference the assembled politicians dispersed. The spring parliamentary recess is traditionally the time for ministers to unwind and reacquaint themselves with their families.

Alistair Darling, the chancellor, spent a couple of days walking in the Scottish Highlands before heading off to Washington for a summit of finance ministers. Labour’s golden couple, Ed Balls, the schools secretary, and Yvette Cooper, chief secretary to the Treasury, were in the Lake District. Ruth Kelly, transport secretary, dragged her brood of four children off to Center Parcs.

Even Brown, who has not had a proper holiday since taking over as prime minister, was hoping to spend a few days with his sons John, 4, and James Fraser, 21 months.

For Brown and his senior colleagues, however, the past seven days have been anything but relaxing. The doubts about the prime minister’s leadership expressed at the Grove kept resurfacing as each new day brought reports of ministerial feuding and embarrassing prime ministerial gaffes.

“It is hard to sit back and switch off when the mobile phone is constantly ringing with tales of Gordon’s latest faux pas or rumours of plotting,” said one senior minister.

The week began with extraordinary claims that Jack Straw, the justice secretary, had “threatened to punch” Balls after a cabinet meeting where they had rowed over who was responsible for youth crime.

Balls remained in the spotlight as his controversial assault on the independence of faith schools began to unravel and cabinet colleagues complained that he was running a closet campaign to succeed Brown as Labour leader.

The economic backdrop continued to worsen with a report from the International Monetary Fund (IMF) challenging Darling’s confident predictions that Britain will escape a recession and further surveys showing falls in house prices.

Brown found himself at the centre of a diplomatic incident when it was reported that he was “snubbing” the opening ceremony of the Beijing Olympics. No 10 insisted that he had never intended to go in the first place, but nevertheless the incident became a metaphor for his indecisive leadership.

Now the already embattled prime minister faces a further blow with today’s Sunday Times poll showing that 62% of voters believe he has a tendency to “dither” as the Conservatives maintain their 16-point lead over Labour, their highest in 16 years.

Even some of Brown’s oldest allies appear to be turning against him. The Sunday Times has learnt that Geoffrey Robinson, the millionaire former Treasury minister whose penthouse in Park Lane was once an unofficial headquarters of the Brown camp, has joined the campaign against the plans to axe the 10p starting rate of income tax.

Tessa Jowell, the Olympics minister, yesterday fuelled the row by acknowledging that the backbench revolt was a “reality check” for the government and not merely the actions of “the usual suspects who will always rebel against the government”.

Brown’s allies will this weekend be pondering ruefully where it all went wrong and asking the more pressing question: can anything be done to stop the rot?

IF the morale of a political party can be measured in the number of inquiries made to the trustees of the parliamentary pension scheme, then Labour is at rock bottom.

“I know of many MPs in marginal seats who have quietly been in touch to find out how much their retirement income would be if they stepped down at the next election,” one backbencher said.

“Others are thinking about retraining to go back into the careers they left when they came into the Commons, such as teaching and social work.”

MPs who retire on grounds of sickness are given special dispensation to claim their full pension before they reach the age of 65. Several have made inquiries about whether it would be possible for them to quit the Commons on grounds of ill health — before voters eject them via the ballot box.

For many that is an increasingly likely prospect. If the results of this weekend’s Sunday Times poll are replicated at the next general election, Cameron would be returned with an overall majority of more than 100. In the Commons tearoom, the talk is of Labour’s “lost generation”, the 100 or so MPs elected in 1997 on the back of the Tony Blair landslide who are facing the end of their political careers.

“This group have never known anything except success,” said one old Commons hand. “Now the tide is turning they are in a state of panic.”

The root cause of the sudden collapse in Labour’s support was Brown’s decision last October to scrap a planned snap election. Overnight, the man known as the “big clunking fist” for his firm leadership became “dithering Gordon”. Last week’s row over the Olympics was a perfect example of the “dither factor” in action.

When Brown announced he would not be attending the opening ceremony in Beijing, he did not mean it to be a snub to the Chinese. He explained he was instead attending the closing ceremony, where he would pick up the baton for London’s 2012 Games.

However, previous remarks from Brown had suggested that he would be going to the more high-profile opening ceremony. “Britain will be attending the Olympic Games ceremonies,” he said last month.

Reporters naturally concluded that Brown had bowed to pressure over Tibet and downgraded his China trip. The resultant “boycott” story went global. Even Hillary Clinton, the US presidential candidate, got involved. “I wanted to commend prime minister Gordon Brown for agreeing not to go to the opening ceremonies of the Olympics in Beijing,” she said.

“Gordon only has himself to blame,” said a Labour special adviser. “He is terrified of giving straight answers to straight questions so he hides behind obfuscation. He should not be surprised when he is misinterpreted when his language is as clear as mud.”

The erosion of public trust in Brown means it has been harder for Labour to blame Britain’s likely economic downturn on the global credit crunch.

Last week the trickle of data pointing towards a house price slump became a flood. Brown knows that the economy’s woes helped to propel Labour into office in 1997, the Tories never recovering from the recession of the early 1990s. The Tory low point was September 16, 1992, when the pound was forced out of Europe’s exchange-rate mechanism on “Black Wednesday”. It was also the month when, according to the Halifax, house prices plunged by 3%.

So when last week the Halifax said that house prices fell by 2.5% in March, the alarm bells began to ring in Downing Street. Brown pointed to the fact that house prices had nearly tripled over the past decade and insisted that the fall was “containable”.

Yet the fear in Downing Street is not just that a rerun of the early 1990s crash in house prices is no longer a remote possibility, it is also that the government is perceived as being out of touch with voters’ problems, including rising energy bills, higher food prices, dearer petrol, more expensive mortgages and other pressures.

Even Brown’s economist friends are deserting him. For years when he was chancellor he wallowed in the praise of the IMF’s annual reviews of Britain’s economy under his management. Last week it said that the Treasury’s growth forecasts were too optimistic and repeated its assertion that the housing market is overvalued by 30%. If it is right, Brown will be facing the next election with the economy crawling along and house prices falling.

On Friday, Darling appeared to be getting his excuses in first. The world was facing its biggest economic shock since the Great Depression of the 1930s, he said, and while “the fundamentals in the UK are sound”, there were “huge challenges ahead”. If Britain succumbs under the weight of those challenges, both he and the prime minister will blame global events. Voters, however, may not buy it.

THE fallout from the collapse in Brown’s poll rating is now being felt at every level in government and the Labour party.

Discontent continues to simmer in Downing Street after the appointment of Stephen Carter as Brown’s chief political adviser. The arrival of the former public relations boss has irritated Brown’s longerestablished cabal of aides.

Carter has hired spin doctors to refresh the prime minister’s sagging image, although last week’s rumour that he was set to hire Phil Hall, former editor of the News of the World who specialises in “difficult” clients such as Heather Mills and the disgraced motor racing boss Max Mosley, has been denied.

The most serious splits are within Brown’s cabinet, where ministers are increasingly pulling in different directions in the absence of clear instructions from the top.

Balls, for example, has been playing to Labour’s core supporters by watering down Blairite plans to increase parental choice in schools. By contrast Alan Johnson, the health secretary, has been pushing ahead with Tory-style plans for patients to be issued with healthcare “vouchers”.

“We just get on and do our own thing,” said one aide to a senior cabinet minister. “We do not really know what Gordon wants so we have given up trying to second-guess him. So we follow our own agenda.”

Fights over patches of Whitehall turf are fast turning into all-out warfare. Balls, Brown’s former chief adviser and protégé, has been accused of using his wide brief as secretary of state for children as an excuse for a wider land grab.

Both Balls’s and Straw’s spokesmen have denied that the justice secretary threatened physical violence towards his colleague — an incident which is now the subject of a Toryinspired internet video game called A Kick in the Balls — but insiders say it “speaks to a higher form of truth”.

“Ed is obsessed with getting his name linked to any announcement that has even the slightest connection to people under the age of 18,” said one disgruntled minister. “He really ought to stick to his knitting, raising school standards and improving discipline.”

Balls’s interests go far beyond the school gates to include the problems of teenage antisocial behaviour and violent video games. “Ed’s people put round a memo suggesting games consoles should be forced to have something called a ‘time out’ button allowing parents to ration their children’s gaming,” one MP said. “He was politely told that we couldn’t really go bossing Sony on how to design their PlayStations and in any case there was already such a button — it’s called the on-off switch.”

Balls has been winning support on the back benches. In the past month alone he has launched the East Midlands local elections campaign in Lincoln and joined colleagues campaigning in Haringey, Harrow, Ipswich and the Wirral.

One of his supporters said: “Someone at Ed’s level could just fill what spare time they have at meetings in London. But every week, as soon as he’s free of Westminster duties, he’s off doing constituency visits. He’s getting a lot of credit for it with the backbenchers.”

Many insiders, however, point to a more cynical motive. The 41-year-old is, according to colleagues, desperate to raise his profile to become the leading “real Labour” candidate should Brown be forced to stand down.

Balls’s spokesman insisted that he was simply focusing on his ministerial duties: “This is just mischievous gossip being spread by the Tories during the parliamentary recess.”

Balls is not the only minister appearing to have an eye on a future leadership contest. James Purnell, the ever so slightly self-regarding work and pensions minister usually regarded as a right-wing Blairite, has been careful not to alienate colleagues in the party. He has watered down his policy on single mothers who claim benefits to appease Labour MPs worried that a clampdown could drive vulnerable families into poverty.

There are also persistent rumours that Charles Clarke, the outspoken former home secretary, might mount a leadership challenge as a “stalking horse” in the hope that it might trigger other candidates to enter the fray. Although Clarke has attacked Brown as a “deluded control freak”, his friends deny he plans a frontal assault on the leadership.

Straw, a cabinet veteran at 61, has perhaps still not given up his leadership ambitions. He has privately told colleagues that he has grave concerns about Brown’s controversial plan to extend the period of detention for terrorist suspects without charge. If the legislation is rejected by the Commons, Straw, ever the sly tactician, has no intention of going down with it.

He is top of the list of names put forward as a “caretaker” leader should Labour grandees decide they have to emulate the defenestration of Thatcher.

Some insiders claim that Labour has been traditionally unwilling to eject faltering leaders. “The right have always been better about hanging on to power,” said one senior party figure. “They have never had the qualms that we have had about being brutal about these things.”

Up to a point. It should not be forgotten that Blair’s resignation was hastened by the “balti house coup” of 2006 when Brownite MPs urged him to step down. It would be ironic if Brown were to be ousted in a similar fashion.

THE weeks ahead will offer no respite for Brown. His enemies are prepared to pounce if the May 1 local and London mayoral ballots bring bad news. The prime minister is left hoping that Ken Livingstone, his former sworn enemy, will pull back the opinion poll lead of Boris Johnson in London, the most prominent contest.

The hostile Compass group is planning a meeting of Labour MPs five days after the vote. Cryptically, the organisers say that the agenda will “depend on the outcome of the election”.

Allies have urged Brown to launch a pre-emptive strike and make preparations for a May 2 cabinet reshuffle.

“Airy talk of relaunches or new policies is not enough,” said one minister. “The only thing that will persuade voters he is serious is blood on the No 10 carpet. Obviously it is high-risk, but at least it will distract attention from the London election.”

The cautious Brown will be reluctant to wield the knife. However, Darling and Des Browne, the defence secretary, could be casualties and Cooper and Caroline Flint, the housing minister, could expect more prominent roles.

Even if Labour avoids the catastrophe on May 1 predicted by the polls, Brown faces other dangers. The Commons debate on his anti-terror legislation is expected next month. Geoff Hoon, the chief whip, has warned Brown that he will be defeated by up to 30 votes on the 42-day issue, unless he makes a concession to the rebels.

In the current febrile atmosphere, if Brown goes for the “nuclear option” and makes the issue of a vote of confidence in his leadership, he could not automatically expect to win.

Then there is the keenly expected return of a ghost from Brown’s past: Lord Levy, Blair’s former chief fundraiser.

Levy’s biography, to be published next month, is expected to be withering in its criticism of Brown. “Michael \ was a confidant of Tony’s during his most explosive rows with Gordon,” said a former No 10 aide. “He has long held the view that there are psychological issues with Gordon.”

Levy, who never drew a salary for his government work, has declined to show the manuscript to the Cabinet Office committee that vets the memoirs of ministers and paid officials. The Brown camp has had no prior warning about allegations in the book.

Brown’s friends are fond of saying “never underestimate Gordon”. They claim a successful trip to the United States this week, where he will see not only President George W Bush but also the presidential hopefuls John McCain, Barack Obama and Hillary Clinton, could do wonders. As one MP said: “If Brown, like Blair, becomes big in the USA, then maybe we in Britain will have to take him more seriously.”

It is some hope. Brown will be playing second fiddle in Washington to a leader who has his flock right behind him and has never been accused of dithering. Pope Benedict XVI is making his first trip to America as pontiff and the Brown visit is secondary in White House minds. The prime minister will be left wondering if there is anywhere he can escape his woes.

Additional reporting: David Smith

What the voters are saying

- If Gordon Brown is to win the next general election, he must convince people like Emma Heath that he deserves their vote, writes Roger Waite. The 35-year-old mother of two lives in Dartford, one of the Kent “super-marginals” that are crucial for Labour hopes. In October, just before the “election that wasn’t” she told The Sunday Times that she was undecided who to vote for. The past six months have made up her mind, she says.

“The situation is terrible at the moment with all the financial problems everyone is having and there’s only one person who I hold responsible and that’s the prime minister,” she said. “The Conservatives will definitely get my vote this time and the main reason is the government’s bad management of the economy.”

- Leslie Powell, a 48-year-old banker, lives in Redditch in Worcestershire, another crucial marginal and the seat of the home secretary, Jacqui Smith. She says she has noticed “the small things” such as the increase in the price of petrol and is probably going to vote Conservative. “My opinion of Gordon Brown has changed,” she said.

“I think he’s a bit of a ditherer now especially after what happened with Northern Rock.”

- Karen Baldwin, below, is typical of the small business owners who are feeling the pinch in the credit crunch.

The 49-year-old owns a pet shop and a garden centre in west Wales and says the worsening economy has hit her directly. “We’ve had a decrease in sales at our garden centre this year for the first time in the 21 years we’ve owned it. The financial situation is very much having an effect on people. It’s something I will look at seriously when I think about where I’m going to place my vote. I might vote Lib Dem or Plaid Cymru but I won’t vote for Labour.”

Times Recommends

• Third of Muslim students back killings
• Curfew tames feral yobs of Cornwall
• £3,000-a-month drug free abroad