David Cameron’s inner circle have spoken of the “DD problem” ever since he gave his defeated rival a seat in his Shadow Cabinet. Never for one moment did they believe that David Davis would detonate like this.

Few if any of the guests at Mr Cameron’s summer party had the slightest idea that their host had just received one of the biggest shocks of his career as he circulated among them in the back garden of his North Kensington home on Wednesday night.

Addressing the throng over a PA, he said that he wanted to keep the noise down, “because I’m not sure the neighbours are Tories”. A more truthful joke might have been that his party in Westminster was far more likely to be about to disturb the peace.

Mr Davis broke the news of his plans to stand down as an MP immediately after Gordon Brown narrowly won the key vote to extend to 42 days the period of precharge detention. The Tory leader, rushing to attend his party, urged him to reconsider and asked him to telephone him later.

Mr Cameron had been under no illusion that Mr Davis felt passionately about the need to resist the Government. Indeed, the Shadow Home Secretary had threatened to resign rather than allow any weakening of the Tories’ position, as The Times reported yesterday. But the Tory leader had no idea that Mr Davis was prepared to sacrifice not just his position but his Commons seat over the issue.

When Mr Davis phoned the Tory leader, Mr Cameron again tried to reason with him, arguing that he would take the spotlight from Mr Brown at a crucial moment. When his arguments failed, Mr Cameron became threatening. He would not provide funds to fight a by-election, he said, particularly since Mr Davis was insisting on complete control over the campaign.

After the call Mr Davis joined a group of Labour rebels for a drink and told them of his intentions. “He was clearly pretty emotional. We all were,” one of those present said.

As news of the decision spread yesterday morning, Shadow Cabinet members expressed amazement. “It’s inexplicable,” one said bluntly.

A friend disagreed. “He’s not a young man, he’s got less cash now than when he entered Parliament. He’s not guaranteed high office in a future Conservative administration. If he loses it would be a shattering blow, but he’d rather go out like this than be slowly overhauled by the Cameroons.”

Until yesterday Mr Cameron was proud of how he had handled Mr Davis. He appointed him as Shadow Home Secretary despite convincingly beating him in a leadership election that was not without rancour and dirty tricks, particularly over allegations of drug use by Mr Cameron. The distrust between the two men could easily have led to the defeated candidate’s exclusion from frontline politics. But Mr Cameron chose instead to use Mr Davis’s undoubted talents, helping in the process to heal divisions.

Despite outbreaks of disloyalty, Mr Cameron stood firm behind Mr Davis in public. Above all, Mr Cameron supported him over precharge detention, when others had expressed their doubts. George Osborne and Michael Gove had worried that the Tories were in danger of getting too closely associated with the civil liberties agenda. David Willetts and Andrew Lansley also harboured reservations.

As the case against 42 days strengthened, both Mr Cameron and Mr Davis shared a sense of vindication. But as the week wore on – and the signals from the Democratic Unionist Party became ominous – Mr Davis started to conceive his plan. He told almost no one, but wrestled in private at the end of long days in the Commons.

Some of those seeking to explain his actions cite tiredness as a factor. One Tory MP said: “He’s been working 20 hours a day on this for weeks. He’s always been a bit obsessive, and on this one he’s flipped.” Another reportedly said that Mr Davis had fallen too heavily under the influence of Shami Chakrabarti, the director of Liberty, with whom he has worked closely.

Friends of Mr Davis dismiss the idea that an exhausted man was bewitched by anyone. Although his actions were informed by personal circumstance and taken at a time when feelings were running high, there is, they insist, a genuine political purpose.

“This is about nailing the Tories to the right position on this agenda and resisting those – like Gove and Osborne – who treat the defence of liberty as some kind of tactical feint,” one said. In promoting Dominic Grieve, the most implacable opponent of 42 days on the Conservative front bench, Mr Cameron is seeking to close down talk of splits. In the short term Mr Davis has thus achieved his ambition in stiffening the Tories’ resolve.

The prospect of an independently funded, independently directed by-election campaign will deeply dismay Mr Cameron, but there will be those around him voicing relief. The “DD problem” has solved itself.

Times Recommends

• Oh no, Mr Darling: you have failed
• BBC survives dogfight over Kennel Club expose
• Where the Rosetta belongs can’t be set in stone