One of Gordon Brown’s closest aides resigned after “losing respect” for the prime minister, it has emerged.

Spencer Livermore, No 10’s director of political strategy, walked out because he had become disillusioned with Brown for “bottling” the general election decision last autumn.

Friends say Livermore, one of the prime minister’s longest serving and most loyal advisers, lost faith when the plan was abandoned at the last minute, despite months of careful preparation, because of a drop in Labour’s opinion poll ratings.

“Spencer spent ages working on the election that never was,” according to a Downing Street aide. “He felt that if his advice had been listened to we wouldn’t be where we are now.”

The disclosure comes amid continuing infighting within No 10 as Brownite loyalists fight a rearguard action against newcomers led by Stephen Carter, the prime minister’s recently appointed chief adviser.

It is understood that Carter, the £180,000-a-year former City public relations executive, is planning to hire several more high-profile figures from the private sector – a move that threatens to lead to the departure of more long-serving Brown aides.

The first signs of serious concern about Brown’s leadership are also emerging within the cabinet, following opinion polls suggesting Labour could be heading for defeat at the next general election. Some disgruntled back-bench MPs, fearful of losing their seats, are beginning to speculate about the leadership, with one suggesting Brown should be asked to retire on “medical grounds”. Another described the prime minister as an “albatross in a tartan waistcoat”.

Labour’s crisis of confidence was triggered by a Sunday Times poll last weekend showing that the party was at its lowest level of popular support since the dark days of Michael Foot in 1983. Other surveys conducted in the following days confirmed there had been a decisive swing towards the Conservatives.

Monday’s announcement of the resignation of Livermore, once listed in a survey as Britain’s most powerful gay man, revealed just how far Downing Street had become a battlefield, with Brown’s most senior aides locked in a war for supremacy.

Livermore’s friends said his departure had been hastened by the appointment in January of Carter, the former chief executive of Brunswick public relations. Carter, who met Brown only once before being offered the job, has been a source of mounting envy among Brown’s “long marchers”.

“Spencer got fed up being left out of key meetings and discussions when he had worked so hard to get Brown where he is today,” said a friend.

Tensions were fuelled by the appearance of a diagram in a trade magazine purporting to show who calls the shots on Team Gordon. Carter was shown as the second most important man in Downing Street, just below Brown in the pyramid.

It appears that Carter – or someone close to him – had a hand in briefing the magazine, a development certain to ruffle feathers further.

Carter is not afraid to pick fights with Brown’s most powerful allies. In a recent meeting he is said to have called for the end of Labour’s “class war”, in a clear warning to Ed Balls, the schools secretary, and Ian Austin, the prime minister’s parliamentary private secretary, who delight in teasing David Cameron about his Etonian background.

Carter’s most important appointment is the antithesis of the Labour working-class warrior. Jennifer Moses, an American appointed as an unpaid policy adviser, is a former Goldman Sachs banker who lives in a £10m mansion in Hampstead, north London.

It emerged this weekend that in February 2007 she described as “outrageous” Brown’s proposals to ask immigrants “seeking citizenship to undertake some community work in our country”. In comments posted in an internet chatroom the day after Brown’s speech, she asked: “Does anyone else find Gordon Brown’s comments on immigrants needing to do ‘public service’ before being granted citizenship outrageous? What if a BNP politician had suggested such a thing?” This weekend a Downing Street aide said: “Now the proposal has been explained in its proper context, Jennifer agrees that it will play a vital part in helping community cohesion.”

Supporters of Carter insist that he simply wants to provide structure and discipline to a Downing Street that had become increasingly chaotic.

His enemies predict that his arrogance will be his undoing. One Brown ally said: “Carter may have been given a free rein to do what he wants, but we all know that a rein can quickly become a rope. And ropes are what you hang yourself by.”

The first murmurings of ministerial revolt began following Tuesday’s political cabinet session. Harriet Harman, Brown’s normally loyal deputy leader, accused the prime minister of a failure to communicate with voters. “When you talk about opportunity, nobody knows what you’re talking about,” she is said to have remarked.

Much of the cabinet disquiet centres on the battles that Brown is choosing to pick. Many ministers worry that political capital is being used up quashing back-bench rebellions on issues where compromise might be a better policy.

“We seem to be going to war constantly with the parliamentary Labour party, asking them to vote in ways that make them feel uncomfortable – all to save modest amounts of money,” said a senior Labour figure.

On Wednesday the government defeated a motion against its post office closure programme, winning by just 20 votes. An exhausted John Hutton, the business secretary charged with winning the vote, told rebels that he would quit the cabinet if the government was defeated.

Even the normally ultra-loyal Balls is said to have criticised Brown’s judgment in resisting the Police Federation’s pay demand, just to save £30m. One Labour insider said the pair had rowed over the issue, culminating with Balls swearing at the prime minister.

Labour’s collapse in the polls and the apparent takeover of No 10 by outsiders have left backbenchers bewildered and angry. The gloomiest are discussing the “Anthony Eden solution” – Eden resigned as prime minister in 1957 after the Suez debacle, citing health problems.

One rebel MP said: “Gordon is blind in one eye and there is evidence that the sight in the other one is deteriorating. Increasingly he seems to be missing words when he is reading statements in the Commons.”

The idea that Brown would voluntarily relinquish power is fanciful but illustrates the desperation gripping some of his critics.

Additional reporting: Holly Watt

Times Recommends

• Model judge and his funky new gown
• First review: Sex and the City
• Thatchers clutch at straws after crops fail