Shortly before disaster struck, Gordon Brown called the man in charge of Labour’s campaign in the Crewe & Nantwich by-election to offer some comfort. Although the prime minister knew the polls were bad, he praised Steve McCabe for his efforts in the parliamentary seat left vacant by the death of Gwyneth Dunwoody, the Labour stalwart.

“Whatever happens,” Brown said, “don’t feel you’re to blame.”

Inside No 10, Brown and his aides were preparing for defeat by trying to massage expectations. Yes, admitted one aide, the Tories were going to win; they might even get a majority as high as 7,000. It was a bid to make any lesser result look almost good for Labour.

Brown did not stay up waiting for the votes to be counted and nor did the Conservative leader. David Cameron went to bed at 11pm, hopeful of victory, and set his alarm for 2am. In the early hours he crept through the house in his pyjamas trying not to wake his family and turned on the television.

The news that greeted him stunned both Labour and the Conservatives. The Tories had seized Crewe & Nantwich, turning a Labour majority of 7,000 into a Conservative one of nearly 8,000. It was better than Cameron had dared to hope and worse than Brown had feared – worse even than No 10’s scenario in its attempts to spin the result.

As the scale of victory sank in, Cameron called key aides; then he went upstairs and, unable to contain his excitement, woke his wife Sam to tell her the result. She soon drifted off again but Cameron could barely sleep.

For the first time in 30 years, the Conservatives had taken a Labour seat at a by-election. The next day, addressing cheering crowds in Crewe & Nantwich, the jubilant Tory leader hailed the “end of new Labour”. It was political licence perhaps, but not without some justification.

The once disciplined new Labour machine seemed to have wheels flying off in all directions. Inside Downing Street panic had erupted. Aides lost no time in blaming “the people on the ground” in Crewe, after all, for botching the Labour campaign. They insisted that nobody at Downing Street had played a hand in the notorious strategy to brand the Conservative candidate a toff. “You don’t get folk from No 10 ringing the people on the ground and telling them to dress up in top hats; that’s not how it works,” said a Brown ally.

The usual Friday morning meeting for special advisers was abandoned and Brown ordered an emergency conference call to cabinet members. He urged colleagues to hold their nerve.

While senior members rallied, trying to brush off the defeat as a by-election aberration, backbenchers vented their fear and dismay. The left-wing Compass group of Labour MPs was lacerating. “New Labour is dead,” it declared, accusing Brown’s government of “serial mistakes” and an “inept, negative and poisonous” campaign.

Stephen Pound, MP for Ealing North, said of the Crewe result: “People say it’s a kick in the teeth. I feel I’ve been kicked in another part of my anatomy.”

Even among loyal Labour MPs, patience was reaching breaking point. “Brown has got about three months to turn things around,” said one normally taciturn backbencher. “If he doesn’t, we’re all f*****.”

Graham Stringer, MP for Manchester Blackley, did not even want to wait that long. “I want a senior member of the cabinet to start a leadership challenge,” he told The Sunday Times. “It just needs one of half a dozen people to have the confidence to stand up to Gordon. Then I’m sure lots would follow.

“Without wanting to sound pretentious, the country needs them to do it. The Labour party needs regeneration. People are reluctant to say this in public, but if we are serious about winning the next election – and I am – we need to do this now. We can’t wait any longer. The electorate have made it clear: they have filed for divorce.”

Last night Brown was at home with his family in Scotland, brooding over whether he can recover. The polls suggest he faces a huge struggle, with the Tories establishing commanding leads of the kind enjoyed by Tony Blair before Labour’s 1997 landslide.

If the Crewe swing of 17.6% were repeated nationwide, the Tories could end up with about 480 MPs and Labour with around 90. But local results are an unreliable guide to general elections and the Tories are taking nothing for granted.

The reasons why were all too obvious when Cameron, after visiting Crewe, flew on to Ayr racecourse, where the party’s Scottish conference was under way. North of the border the Conservatives have just one MP. The decrepit band of blue rinses gathered for the Scottish conference were a stark reminder of the hurdles that still lie between him and the keys to No 10.

Outside the conference centre a van advertised a concert by the ageing rock group Status Quo, still going strong. The questions facing the Tories are these: can they maintain momentum? Can they transform what has been the status quo for more than a decade?

AT Conservative Central Office an “implementation unit” is preparing the party for office. Among the team are management consultants, former civil servants, Lucy Neville-Rolfe, the Tesco director, Theodore Agnew, an insurance broker, and a political strategist formerly employed in Blair’s delivery unit at No 10.

Detailed plans are being drawn up for enacting sweeping policy changes in Cameron’s three priority areas: education, welfare and the family.

By the summer the team will be holding serious talks with the civil service, laying the ground for a possible transfer of power. Some policies are already being run past retired mandarins for views on their practicality.

Opposition parties have the right to demand these discussions with mandarins ahead of general elections and such talks have been held before. But Cameron’s inner circle predict that the next round will be different: “In the past the talks have been slightly perfunctory; it all seemed a bit academic. This time it will be serious.”

The mood is one of optimism that a sea change in political attitudes is under way, one with echoes of the shifts that happened the last time the Tories took a Labour seat in a by-election. In 1978 the Labour government of Jim Callaghan was in disarray. Then, as now, voters were worried about inflation, soaring oil prices and immigration. The “winter of discontent”, blighted by strikes, was looming.

At the same time the Tories were developing a new strategy, as Cameron has been doing recently. The Centre for Policy Studies, set up in the 1970s, formulated a Conservatism which appealed to skilled working-class voters. It worked. In the 1978 by-election in Ilford North, a Labour seat, a young Tessa Jowell lost to Vivian Bendall, the Tory candidate. He was an estate agent. It was a foretaste of the aspirational, home-owning Thatcher revolution that began with her first general election victory the following year.

In Crewe last week something similar was in the air. Voters were furious about soaring fuel and food prices and the fiasco over Labour’s scrapping of the 10p tax rate.

David Sullivan, 35, a sales manager, said: “It’s the cost of everything that’s the problem: petrol, food, beer, cigs. The price of petrol is going up every day, it’s crippling people and we all know that 70% of what we pay is tax.

“I think we were all taken in by Tony Blair in the past, but now all the tax increases are coming back to haunt them. I did vote Labour last time but I doubt I will again.”

The Tories ran a powerful campaign, helped by funding from Lord Ashcroft, who has supplied £470,000 in material help and £300,000 in cash already this year, according to the latest filings.

Although Eric Pickles, one of the campaign managers, said the party targeted everyone in the constituency, it was notable that many of those who switched to the Tories were skilled working-class and aspirational 20 to 35-year-olds.

Labour had little to offer except negativity. There was widespread distaste at its attempts to portray Edward Timpson, the Conservative candidate, as an upper-class twit born with a silver spoon in his mouth. A stunt early in the campaign, in which Labour activists donned waistcoats and top hats to mock Timpson – the son of a millionaire who made his fortune repairing shoes – backfired spectacularly.

“I’ve read all this stuff about him being a toff,” said one voter. “But I couldn’t care less. There are more important things to worry about, like being able to pay the mortgage.”

Labour spin doctors in Crewe circulated a photograph of Conservative canvassers delivering party leaflets in a Bentley. They failed to mention that the cars are made in Crewe and residents are proud to have the company, which employs 4,000 people, in the town.

Stephen Baxendale, a retired railway worker, said: “I saw the picture, but so what? They build Bentleys here. Instead of that sort of stuff, Labour should be explaining why things are going so wrong.”

Labour’s desperation was evident when Tamsin Dunwoody, the party’s candidate and daughter of the previous MP, described herself as “a single unemployed mother of five fighting for a job”. In fact she is descended from a baroness and has an entry in Burke’s Landed Gentry.

Who was to blame for this dismal attempt at class war? Some backbenchers are suspicious that Ian Austin, Brown’s chief parliamentary aide, and Ed Balls, the schools secretary, had a hand in it, but their allies furiously deny the suggestion.

Yesterday McCabe magnanimously took responsibility, admitting that the top hat stunt had spiralled out of control: “Those of us working in Crewe were responsible for the way it turned out. There was never a strategy to present Timpson as a toff. We were just poking a bit of fun about the chap.”

Whoever was to blame, the damage was done and it was Labour, not the Tories, which was seen as the “nasty party”.

Brown seemed incapable of rescuing the campaign. After his political cabinet met last week, one senior figure said: “People did not come out thinking they had seen a brilliant masterplan that was going to get Labour out of this.” Another said: “The political cabinet was awful. The tank is empty.”

THE failure laid bare Labour’s loss of direction. What does the party stand for? Who is it fighting for? Its traditional supporters no longer know.

A number of unions went on the offensive, lambasting Brown over his insistence that public sector pay rises must be kept in line with the Treasury’s 2% inflation target as measured by the consumer prices index. They argue that rises in the retail prices index – running at 4.2% – provide a better guide to the cost of living.

A new study by the Centre for Economics and Business Research calculates that basic household spending is rising even faster – by 5.7% a year.

Mark Serwotka, general secretary of the Public and Commercial Services Union, branded Brown “a miserable PM who should hang his head in shame”. Unite, the biggest union, plans to ballot 70,000 local government workers for industrial action over pay. Unison, the largest public sector union, also threatened strikes.

Prospect union, representing 102,000 specialists and managers in the public and private sectors, warned the government that its members’ patience was running out.

Jacqui Smith, the home secretary, was subjected to a ferocious personal attack when she attended the Police Federation conference, which was furious at her refusal to meet a pay claim in full. With a barbed reference to Smith’s admission that she once smoked cannabis, Jan Berry, the federation chairman, said: “Your recent crimes have been more for the serious fraud squad than the drugs squad.”

Alluding to how Balls had stood up for better pay for teachers, Berry added: “Home secretary, what is it that Mr Balls has that you do not?” Smith sat stony-faced.

Senior Labour figures say she is privately seething with Brown, blaming him for the breakdown of her relationship with the police and for landing her with the nightmare task of persuading Labour backbenchers to extend the length of time that terror suspects can be detained without charge.

Big business has fallen out of love with Labour, too. At a dinner last week held by the Confederation of British Industry, disillusioned executives rounded on Labour guests, criticising the party’s handling of the economy.

Even within the higher levels of the party itself, antagonism has burst into the open. Lord Falconer, the former lord chancellor, savaged Brown’s flagship constitutional reforms, labelling them “trivial, containing nothing of significance”, and “designed to meet problems that do not exist”.

The prime minister was so beleaguered that he did not go to Crewe to help the campaign. As polling day approached, Alistair Darling, the chancellor, admitted to friends: “We’ve lost our way.”

In the Commons on Friday, George Osborne, the shadow chancellor, and a shadow cabinet colleague could barely disguise their glee at the scale of the Tory victory. They joked about launching an offensive to steal the seat of Balls, Brown’s closest ally. “Perhaps we should go and visit his constituency next week,” they mused.

Publicly, the shadow cabinet is desperate to avoid any triumphalism. Members know they have a long way to go to persuade the public that the party has policies worth backing. What may swing voters’ views is the economic slowdown and squeeze on household budgets.

Questions of tax and waste are coming to the fore. For the first time in years, tax cuts are on the agenda, even for the Liberal Democrats, long wedded to revenue raising. In a speech at the Policy Exchange think tank last week Nick Clegg, the Lib Dem leader, claimed that Brown had presided over the largest hike in taxation of any government for more than 30 years.

“In real terms we are now taxed over £150 billion a year more than we were 11 years ago,” he said. “Governments must never tax for the sake of it. The Liberal Democrats will focus all our attention on cutting taxes – from the bottom.” Boris Johnson, London’s new Tory mayor, has made cutting waste a central plank of his regime, hiring Tim Parker, the former AA chief executive – dubbed the “prince of darkness” for his record of cutting jobs – as one of his deputies.

Nationally, the Conservatives are more cautious, although Cameron delivered a speech last week entitled “Living within our means”. Osborne is determined not to be drawn into making promises that might be impossible to keep.

One shadow minister said: “It would be crazy to make the sort of detailed pledges some people want. We would not be a party ready to govern if we did that. We can’t tie ourselves to a budget two years ahead of when we might be in power.”

Brown and his supporters cling to the hope that an economic revival by spring 2010, the latest point at which a general election can be held, will save their jobs. They insist that the prime minister’s woes are inextricably tied to the downturn.

“We are going to come through this thing and confidence in our ability to handle the economy will be restored. If we get through this, we will actually emerge stronger from it,” said one of Brown’s closest allies, with dogged optimism.

Anthony Seldon, headmaster of Wellington college and Blair’s biographer, also feels it is too soon to write off the prime minister. “I don’t think the by-election is going to be the start of a Tory triumphal march down Whitehall,” he said.

“He’s got nine months to pull it around. Look where he was nine months ago. We could suddenly find ourselves asking how we ever thought David Cameron was the person for high office.”

Many Labour MPs are far less sanguine. They believe Brown is fatally damaged. Although John Major was able to make a comeback in 1992, he was popular even as his party was not. Brown’s personal poll ratings are horribly low.

Yesterday one aide said: “The problem with Gordon is nobody likes him. If you make so many enemies on the way up, not many people will come to your rescue on the way down.”

Mark Wickham-Jones, professor of political science at Bristol University, believes leaders in Brown’s position are doomed. “It’s incredibly serious for Brown now. Things have snowballed,” he said. “He hasn’t got the charisma to turn it round.

“The only way back for Labour is if he takes a trip to a sanatorium and resigns on grounds of ill health.” THIS weekend Charles Clarke, Alan Milburn and Stephen Byers – all cabinet ministers under Tony Blair and figures considered likely to stick the knife into the prime minister – were lying low.

David Miliband, the foreign secretary, is also playing a waiting game. He is seen as a Blairite contender to replace Brown, but this weekend denied any thoughts of challenging his leader.

James Purnell, the work and pensions secretary, displays similar discretion but it is notable that he has hired Philip Collins, a former speechwriter for Blair and an arch-critic of Brown, as an adviser.

In the latest issue of Prospect magazine Collins makes a devastating attack on Brown’s style of government. “Labour is failing to win – or even to grasp – the big political argument: how to ensure people are in control of their own lives,” he writes. While Brown remains obsessed with central control, he says, Labour must “liberalise or die”.

One former minister who wants the prime minister to step down estimates that at least 120 backbenchers privately want to see Brown go. A leadership challenge would require at least 70 MPs to come out publicly against Brown, but insiders say an informal move to persuade Brown to step aside is far more likely.

Jack Straw, the justice secretary, is said to be “seriously considering” urging Brown to step down for the good of the party, although he publicly denies it.

The Tories hope that the prime minister will stay, wounded and unpopular, while they get on with building a credible party of government. For the first time in years it seems a genuine possibility.

As Cameron was moved to say: “We are beginning to win the battle of ideas in the way the Conservatives won it at the end of the 1970s.”

Additional reporting: Philip Cardy

How the Tories aim to change Britain

TAX
Cameron and Osborne are still ultracautious about promising income tax cuts. Flagship promises so far are scrapping stamp duty for fi

rst-time buyers on homes up to £250,000, and increasing inheritance tax threshold to £1m. Tories would also impose a £25,000 levy on non doms and cut corporation tax

HEALTH
The party scrapped plans to offer patients vouchers to go private. New focus is on cutting red tape and hated Whitehall targets. Doctors like the idea, but critics warn any let-up in pressure from the top could lead to longer waits for patients

WELFARE
Crackdown on incapacity benefit, cutting payments by £20 a week in a “tough love” approach. All 2.64m people on sick pay forced to undergo “fit to work” tests. Increase in working tax credit for couples

EDUCATION
Tories are such strong supporters of Tony Blair’s city academies that they want to recruit Lord Adonis, the Labour minister in charge of the scheme. Swedish-style reforms would allow parents, charities and philanthropists to open schools, while money would follow disadvantaged pupils

IMMIGRATION AND CRIME
Plans for an Australian-style points-based immigration system. Have called for an annual cap on immigration. Will scrap identity cards, on grounds that they are expensive and won’t prevent terrorism. Big prison-building programme

FAMILY
Plans for increased maternity and paternity benefits

Key influences in David Cameron’s team

MICHAEL GOVE
A key member of the “Notting Hill set”, Gove was promoted to the shadow education brief although he was only elected in 2005. A former Times journalist from a state school background, he epitomises the meritocratic ethos Cameron wants to foster

NICK BOLES
Co-founder of the infl uential think tank Policy Exchange, and currently acting as Boris Johnson’s chief of staff. Educated at Winchester and Oxford, Boles, 43 and openly gay, has been selected as the Tory candidate for Grantham and Stamford

FRANCIS MAUDE
Although he stepped down as its chairman, Maude remains a key moderniser eager to broaden the party’s appeal. Now in charge of preparing how the Conservatives would implement policies if they win power

STEVE HILTON
Cameron’s director of strategy has a low profile that is in inverse proportion to his influence at Conservative HQ. An expert in creating “brand awareness”, he has a say in policy as well as presentation

Feeling your pain

When Gordon Brown started insisting that he “feels your pain” he entered a strange universe inhabited by Bill Clinton and Christina Aguilera. These days it is not enough to sympathise with people – to understand another’s situation. Politicans must strive to empathise – to enter the internal world of another person and see the world from their point of view. In 1992 the former US president Bill Clinton was skewered by an Aids campaigner saying:”We’re not dying of Aids as much as of 11 years of government neglect” Clinton stopped him in his tracks by saying: “I feel your pain.” When Bob Dole quit the US Senate to run as a Republican presidential candidate he went one better: “I understand. Some would say, ‘I feel your pain.’ I want to cure your pain.” The last word must go to pop singer Christina Aguilera whose song “I feel Your Pain” may well be on the PM’s ipod. “Forever in the sky and even in the rain/ We will last forever/ I will not forget, I feel your pain”

Good riddance Gordon: how to change the leader

The rules on replacing a Labour leader are, perhaps not surprisingly, a bureaucratic nightmare and depend on whether there is a vacancy or a challenge. Anyone nominated to challenge an incumbent requires the support of 20% of MPs. That equates to 71 of Labour’s current 351. The candidate must inform the general secretary of the Labour party (a position vacant at present) that they accept their nomination at least two weeks before any voting begins. Nominations are printed in the agenda of the party conference, and a contest can proceed only if there is a majority vote in favour of it. If a contest proceeds, MPs get one third of the vote, members of the Labour party another third and the trade unions a fi nal third. If the leader has stepped aside and there is a vacancy, the threshold for challengers is lower: each nomination must be supported by 12.5% of the party’s MPs.

Contenders to replace Brown

JACK STRAW

Strengths: a steady hand on the tiller – just what’s required when the ship of state has turned itself into the Titanic. The justice secretary has support from all wings of the party

Weaknesses: though an energetic gym bunny, he’s 61 and might be seen as too old for the top job. Labour won’t want a Ming

DAVID MILIBAND

Strengths: young, ambitious, plenty of hair and looks nice on television

Weaknesses: can he convince enough Labour MPs to share his obvious delight in his own ability? Tends to lapse into think tank jargon

ALAN JOHNSON

Strengths: the former postman who delivers. Is universally popular with Labour MPs. Unlike many ministers, he appears to be a proper human being

Weaknesses: does he really want the job? Once said on Desert Island Discs that he wasn’t cut out to be prime minister. Mind you, lack of talent hasn’t stopped some we could mention

JAMES PURNELL

Strengths: cultured and smart, the youthful work and pensions secretary has the sort of easy presentational appeal that Brown so obviously lacks

Weaknesses: few people outside Westminster have a clue who he is or what he does. Which doesn’t stop him being smug

ED BALLS

Strengths: Gordon’s closest adviser. Has been an energetic children’s secretary

Weaknesses: Gordon’s closest adviser. Perhaps even more pleased with himself than Miliband. Some MPs find him so annoying that they admit wanting to punch him

HARRIET HARMAN

Strengths: driven by Hillary Clinton-style self-belief, she confounded critics by winning the race for deputy leader. Outclasses Brown at the despatch box

Weaknesses: very divisive. Wins friends, but produces fury in others. Appears bossy, toffish – being called Harriet doesn’t help – and too self-interested

Times Recommends

• Margot Fonteyn: In her footsteps
• Forget beautiful people, let's have wonky Brits
• Fundamentalist brew: faith and power