The parliamentary Conservative Party departed Westminster for the summer recess this week in a fit of the jitters. These jitters are entirely self-induced.

For there can be no Conservative crisis unless the party talks itself into one. The political argument has not changed: Tory opportunities remain for the seizing. The party’s estimate of Gordon Brown’s weaknesses remains accurate. The potential strength of a steadily emerging Tory platform and the underlying appeal of David Cameron’s pitch to a 21st-century British public is still there: a bonus that none but he could have brought. Unless Mr Brown calls a general election very soon, I still believe Labour should lose next time.

For the Tories now to run gibbering for cover after striking one false note in a speech about grammar schools, not doing as well in a by-election as they had hoped, being caught by the weather after their leader had arranged a trip to Rwanda and experiencing in the polls the flipside of a new Prime Minister’s predictable honeymoon would look unforgivably weak. Even Mr Cameron’s fiercest rightwing critics will, if they are honest, have to admit that Mr Brown would relish the chance to describe the Tories as having lost their nerve and lurched back to their old prejudices.

So the case for a closing of ranks this summer and during the conference season is urgent and strong. Let me say what I think is behind the minor but dangerous panic attack. Two men and a particular type of right-wing media commentary are involved. Neither man is David Cameron, whose judgment has misfired less often than that of his predecessors, and whose steadiness under fire has been extraordinary. No, the causes of instability lie outside him.

One of the men is Gordon Brown, the other David Davis. Mr Brown is having an easy early run and his supporters think – I believe wrongly – that it can continue. Mr Davis has behaved impeccably. The Davis problem arises simply from his being there, and being impressive.

Time and the tide of events will solve the Brown problem. Today much of the media is feeding from his hand, helping to produce an impression of dynamism and decisiveness. The impression is false. Mr Brown is turning out to be, as I suspected, a better spinner than Tony Blair.

This administration’s response to the recent flooding has been absolutely routine and nothing to do with the Prime Minister. In Iraq we are in the same hole we always were, and still digging. One interview, half-disowned, from a semi-house-trained junior Foreign Office Minister in the House of Lords about a rebalancing of relations with Washington does not amount to a rebalancing. In Afghanistan our Armed Forces are getting deeper into difficulty. Mr Brown is fiddling around a bit with the structure of the health service and the status of city academies. He has talked big on some rather small changes of a constitutional nature. And he has signalled government intervention in the housing market on a scale, and with results, that he cannot possibly deliver.

And that’s it. One cannot reasonably expect more within the first month of a new premiership. If you believe Mr Brown has a range of big ideas for government in his locker – ideas for substantially improving public services without increasing taxes – then you are entitled to hope that these will mark him out as a winner when we see them. But we have not seen them yet. So far George Osborne’s analysis, as Shadow Chancellor, of the micromanaging and macroconfusing nature of Brownite thinking, holds good. As the dawn mist clinging to a new premiership burns off, these defects should be exposed.

The Davis problem can be solved only by Mr Davis himself. This summer he needs to go one step beyond loyalty (already displayed) and convince his potential supporters that the future of the Conservative Party, its Centre-Right as well as its Centre-Left, and perhaps the future of Mr Davis himself, lies in the success of Mr Cameron’s political project. To every mutterer, a knockdown answer should be available: “Apart from David Cameron, no imaginable leader is available.” Only Mr Davis can really convince his admirers that this is true.

And the right-wing commentary? Rather like that harbinger cat recently discovered in an American nursing home for the elderly (the creature’s habit being to curl up on the beds of those residents who are not far from their final hour) a group of rightwingers, reactionaries and xenophobes styling themselves as Tory columnists has been acting as a false friend to the party, drawn towards everything that is morbid within its body politic, sucking it towards its grave. Who knows (and I doubt) whether the voters take much notice, but the parliamentary Conservative Party does.

Whether the cat is simply drawn towards death or helps to induce death is unclear, but what is not in doubt is that the cat is associated with death. Residents desirous of a continuing lease of life are advised to hear the purr as a death-rattle and kick the creature from their blankets. Likewise with these commentators. Curled up on the end of the party’s bed, purring their reactionary opinions and their evident distaste for Mr Cameron’s leadership, they spell trouble. Locked in each others’ arms, a reactionary Conservative Party and a Fleet Street philosophy whose unspoken refrain is that the best years of all our lives lie behind us and that 21st-century Britain is a most undesirable place, could, if they were allowed to, take each other tangoing together into the eternal political night. These writers are not in any useful sense on the Conservative Party’s side.

This really isn’t a Left-Right thing. For what it’s worth, I am a Eurosceptic and I think Mr Cameron is too. I am bored with Team Cameron’s stunts and I hope that soon he will be too. I am certain the party should be talking more about taxation and the size of the State. And I agree with those who say that greenness and coolness and inclusivity are all very well, but where are the messages that will chime with the Middle England that I think I recognise in the Midlands? These are arguments we need to push forward within the Conservative Party. They can be won.

But none of this will be achieved without power, and the first leader in a decade to bring the party a sniff of that power, to make winning believable, has been Mr Cameron. For 50 or 60 ageing Tory politicians with safe seats beneath their bottoms and familiar bees beneath their bonnets, to sabotage a rescue that has only just begun would be an act of such idiocy as to amount almost to a release. We could then leave the Conservative Party to its fate. For any true Conservative to raise his hand against the leadership now would amount to complicity in the attempted suicide of the entire party.

Times Recommends

• Caesar & Cicero: Balls & Mandelson
• The anti-sex brigade are the worst
• Small act of charity won’t change the world