When Gordon Brown set off for his trip to the Gulf States last weekend, whatever other uncertainties might have been playing on his mind, he was relatively sure of one thing: unless Labour's private canvassing was horribly wrong, the party was on track to win the Glenrothes by-election.

The Prime Minister was privy to the results of a mass canvass completed towards the start of last week which showed that voters who had previously said they were “don't knows” were heading for the Labour camp by a margin of three to one as Thursday's polling day approached.

The results of that canvass were kept to a group of five party top brass: the Prime Minister, Jim Murphy, the Scottish Secretary, Iain Gray, the Labour leader in the Scottish Parliament, Gordon Banks, the Glenrothes campaign manager, and Colin Smyth, the secretary of the Scottish Labour Party. They made a pact to tell no one else and, other than to say that the result would be “close”, were content to sit back and watch as the premature claims of victory from Alex Salmond and the SNP grew louder and louder, media pundits busied themselves estimating the Nationalist majority, the bookmakers installed the SNP as odds-on favourites and even Labour ministers and backbench MPs at Westminster prepared their defence for what they felt was certain to be another by-election defeat for Brown.

Early today, as the votes were being counted, Labour's by-election high command knew that unless thousands of Fifers had been telling lies, Mr Salmond was in for a shock.

The sucker punch landed with such force that it left the SNP leader and his party filled with doubt for the first time since they came to office at Holyrood in May last year.

Mr Brown, whose Kirkcaldy and Cowdenbeath constituency borders Glenrothes, kept in daily touch with events on the ground and broke with prime ministerial tradition by twice visiting the seat. His wife, Sarah, knocked on thousands of doors and gave a running commentary on what she found.

The Prime Minister also made sure that Labour would not want for forces on the ground, and by polling day there were 800 Labour activists in the constituency.

A key Labour strategist told The Times: “No one should underestimate Gordon's role in this. He knows Fife like the back of his hand, knows how the people here think and he felt all along that this by-election could be turned in our favour - though I think even he will be a bit surprised that we did it so easily.”

So what did go wrong for the Nationalists? The answer lies in one word - incumbency. The SNP may be in power in Edinburgh and on Fife council, but in a campaign dominated by local issues, it was there to be shot at. Labour early on seized on the issue of increased home care charges for the elderly imposed by the council, led by Peter Grant, the SNP candidate, and did not let go.

It was, as the SNP alleged, negative campaigning. But as past masters at such tactics the party should not have been surprised that it became the dominant issue.

It tried hard to answer the Labour accusation that the council was forcing the vulnerable elderly to pay unreasonable amounts for items such as home alarms but never managed to neutralise the Labour propaganda.

Labour, in government at Westminster, fought this campaign as the opposition in Scotland and it paid off. Then there was Lindsay Roy, the Labour candidate, a personal friend of the Prime Minister and Mr Brown's own choice to fight the by-election. Mr Roy, a 59-year-old headmaster, came across as a throwback to a bygone age, appearing to wear a dazed expression as if surprised to find himself on the stump. This was misleading. He has won praise from parents and colleagues over the years for his teaching and his talent for communication. He is popular locally and garnered a huge personal vote.

While the campaign concentrated almost exclusively on local concerns, it was played out to the doom-laden background of the global financial crisis and Mr Brown's role in handling the fall-out.

Fifers are notoriously clannish and took pride that one of their own was being praised around the world for his role.

The bail-out of Scotland's two leading financial institutions, HBOS and RBS, raised questions at the back of their minds about whether Mr Salmond's independent Scotland would be able to do likewise.

Last night, Mr Salmond, surveying the wreckage of the SNP's campaign, admitted that it was his fault. For a politician who has appeared to walk on water for the best part of the last 18 months, this is not a position with which the First Minister is familiar.

“The failure is of the campaign leadership - which is me effectively - for not recognising that we should have changed our campaign to face down a scaremongering campaign,” he said: “That's my fault for not having my finger on the political temperature.”

Times Recommends

• Donor organ allowed child’s own heart to heal
• Michael Jackson: Not the final chapter
• Brüno toned down for 15 rating