A “vacuous coward” is how Gordon Brown was described to me recently by one former Cabinet minister who has what we shall politely call doubts about the Chancellor’s leadership abilities. Nothing much to report in that: any hack in Westminster knows how to find someone who dislikes Mr Brown and will denounce him.

Far more important than the few MPs who are absolutely sure that the Chancellor is not the man to succeed Tony Blair is the great silent swathe who are uncertain that he is the right man, but do not know what to do about it. Mr Brown has not won them over; he has merely silenced them. The super-Brownites, the Browner than Browns, the Bruns, are far fewer than the noise they make would suggest: more bruit than Brun.

It is through these nervous spines that a shudder will have run at an opinion poll in The Guardian yesterday predicting a clear victory for the Tories if David Cameron meets Mr Brown at the next election. Were the ICM prediction — 29 per cent Labour, 42 per cent Conservative — to be correct, dozens of Labour MPs would lose their seats and the Tories would have a working majority. No Labour Government; no hung Parliament; a full-on Tory Government. The “eeks” heard in little corners of Westminster yesterday were not emanating from the Commons’s plague of mice.

The public will decide the contest for the Labour leadership. Research conducted for the party shows that its next leader must be able to appeal to new groups of voters in the shires. “If Labour won every single seat in London, Scotland, Wales and the six metropolitan counties,” the research cautions, “we would still be 36 seats short of a majority” at the next general election.

There are three core groups that Labour will need to win to gain victory under redrawn boundaries. Three clusters of seats represent the groups. The first is the M4 corridor: Labour must win 11 seats in and around Bristol, Gloucester and Swindon, typified by the new constituency of Filton & Bradley Stoke. “Its dominant characteristic is a huge proportion of young families with mortgages, working in the private sector in technical white collar or highly skilled manual work,” says the internal research.

The second target group is around the M25 corridor: Labour gained 22 seats in the counties adjoining London and the M25 in 1997. It still holds 15 but most are marginal. Typical of them is Harlow, with a majority of just 97. “Like other seats around the fringes of London, Harlow is an area with some Labour history, but it voted for the Conservatives in 1979,” says the research. “It is a straight Labour-Tory fight where the Lib Dems got 13 per cent of the vote in 2005. It has lots of self-employed workers with small businesses, dependent on a stable economy and good local schools.”

The third cluster is along the M1 corridor, “a symbolic front line where almost every seat is marginal”. Before 1997 Labour was invisible in the area, but it then gained five out of six seats in Northamptonshire, two in Milton Keynes, two in Luton and MPs in Bedford, Rugby and Warwick. “Labour’s front line contains seven seats in and around Milton Keynes and Northampton and others which the party must try to regain, many of them with an industrial and working class core but which are now characterised by rapid expansion, the development of service industries and high-tech work, the kinds of communities which were off-limits for Labour in the 1980s.”

These are the communities that Labour’s next leader needs to persuade MPs and party members that he can win over. Nothing is set in stone about the Chancellor’s succession; it would take only a late challenge from, say, Charles Clarke, to blow the contest wide open to anyone. Many ministers, in and out of the Cabinet, are uncertain about Mr Brown. MPs in marginal seats are terrified. Labour members are far from enthusiastic. Even political journalists are nervous, knowing the ugly mix of flattery and coercion that is the modus operandi of the Brown team. You might be flattered and invited to tea today, but that only means you could be bullied and locked out tomorrow.

And it is around the equally uncertain figure of David Miliband that all the doubts are coalescing. Mr Miliband is probably the most uncertain of all: uncertain whether to stand (ignore his protestations to the contrary), uncertain whether to do so would split the party and destroy his own future under Mr Brown. Uncertain, I imagine, whether it might not be better for him to wait and go for the leadership if Mr Brown loses the next election, rather than take a wild punt on it now.

Would Mr Brown lose the next election? Some Labour people are convinced he would. I am pretty sure he could beat Mr Cameron: his gravity will play well against the Conservative leader’s lightness, and any possible Cameron Cabinet looks ridiculous against the potential Labour one with all its experience, not least at running the economy. The Tory revolution is barely more than a one-man band. Labour has not begun to exploit any of this yet.

But whether Mr Brown can defeat Mr Cameron is not the question. The question is whether Mr Miliband is more certain to do so. The public barely knows Mr Miliband. He is a relatively junior Cabinet minister, with complex political views, far from the right-wing Blairite he is lazily branded. Put under scrutiny, his winning patina might turn out to be just that, a mirage. And although his youth looks like an advantage (imagine how old Mr Brown will look if Menzies Campbell is replaced by the 40-year-old Nick Clegg before the next election), it could also remove Labour’s strongest card — experience — against the youthful Conservative.

I am not saying Mr Brown is in trouble. I am saying he easily could be. I am saying listen to the silence.

Times Recommends

• Fragility always gets my sympathy vote
• Double it and you get the real jobless total
• Has al-Megrahi paid his council tax?