A stranger approached me at St Pancras station on Thursday, her tone more inquisitive than hostile. “Why do you hate Gordon Brown so much? Your ferocity seems personal. Has he hurt you somehow?”

What I told her, which is true, only deepened her perplexity: that I have no score of any kind to settle.

Taking leave of her and walking to my train I resolved to think harder for the answer. She deserves it because I intend today to return to the attack with more ferocity than ever. I sense for the first time that Mr Brown may be forced to quit.

But why my special fury? Allow me first to discount some of the attacks people make on Gordon Brown's suitability for the top job: unfair attacks.

I don't think it matters that he's dour. I blush for him at advisers' ham-fisted attempts to make him look cuddlier. The British are fair-minded people and know very well that to be shy - even dull - does not make a man worthless. We do not expect every leader to be a prancing show pony, and after the last one we were ready for still waters that ran deep. “Not flash, just Gordon” resonated.

Nor do I think it matters if he's obsessive. Great leaders sometimes are. Narrow focus often accompanies high intelligence, and - again - the British are mature enough to see strength as well as weakness in a blinkered mind.

Nor do I care if he's rude. I've heard the tittle-tattle about red-mist rages, phones thrown at walls, secretaries reduced to tears. But again, so what? We're not choosing a flatmate: we're looking for a leader.

Great leaders are sometimes brutes. “Psychologically flawed”? So was Churchill. I do think Mr Brown is pretty weird, but it doesn't bother me.

Finally, though unsettled by his “Macavity” reputation for avoiding crime scenes, and by the “ditherer” tag too, I see neither criticism as fatal. Hesitation can be wise; and leaving colleagues to take the rap when things go wrong is not uncommon among ruthless men.

No, for all I care, Mr Brown can be a bean-counting, flak-ducking, procrastinating, tunnel-visioned, trainspotting monster. These are human qualities. I like human qualities. It's vacuums I despise. What is unforgivable is the empty space in Mr Brown's head where an idea ought to be. One big idea, one bold, brave, all-consuming purpose, one gripping sense of political direction, would outweigh all the carping criticisms we may have of Brown the man.

But where there should be thought, there is only calculation.

Forgive my returning to a column I wrote here in September 2005. Recounting the story of the Wizard of Oz, I reminded readers that when the great curtains in the Emerald City were finally pulled back, Dorothy and her friends found... nothing. Just a small, anxious figure who had until then been able to project a big, confident illusion. The (then) Chancellor of the Exchequer, I submitted, “may be hiding a dreadful secret: that he has nothing to hide, nothing to propose”. At the time only George Osborne (already Shadow Chancellor) seemed to see the nakedness. Almost to a man and woman, my colleagues in the media, Mr Brown's army of admirers in the Labour Party, and many of the parliamentary Conservative Party too, were describing him as a master strategist, towering intellect, Cabinet colossus and political titan whom we underestimated at our peril.

I just didn't see it. I had been watching him for years. I saw a serious-minded, thorough and quite scholarly man - a capable swot - but had never heard him say anything remotely courageous, interesting or new. From his table-banging, shapeless speeches not the ghost of an outline of a distinctive political philosophy could be discerned. In interview he was defensive. On the rostrum he was blustering. In print he was opaque. And time and again the claims he did make didn't stack up: there was niggling, small-scale dishonesty in the way he used facts and figures. Watchful, prickly, aggressive, yet in some strange subterranean way (I guessed) scared, I saw in him a paralysing failure of intellectual confidence: a yawning absence of creativity.

What was not absent was ambition. A craving to be boss consumed him. And the result was a bike-shed-style bullying of his parliamentary party: “Join my gang because... uh... we're going to win, and if you're not in the gang you'll get a kicking.” They did join, and now he's Prime Minister.

The failure of senior colleagues to rumble Gordon Brown earlier will stand, for the Labour Party, as an enduring disgrace. Many who knew better bit their lip as their party shuffled towards a wedding they feared would end in tears, but nobody dared to oppose. Mr Brown's leadership became inevitable only because nobody had the guts to query its inevitability. The marriage banns were read, and in the pews, as anyone who knew just cause or impediment why this man should not be Prime Minister was exhorted to declare it, feet shuffled but nobody spoke.

Huge question marks have been placed against the moral courage of two or three key figures in today's Cabinet. They don't have long to remove them.

But bandwagons within British parliamentary politics are compelling things. Less explicable has been the collaboration of most of the senior echelons of our media class in the myth of the Wizard of Kirkcaldy.

Not once in the past decade have I heard a media colleague explain with any force or clarity what the Brown vision for change amounted to; hardly ever, however, did I hear a senior colleague seriously question that he had one.

Which brings me to what I should have confessed to that lady at St Pancras. At the root of the vehemence of my attack on Mr Brown, is not Mr Brown at all. Mr Brown is negligible, an emperor with no clothes. What has been maddening this past decade has been to watch the courtiers praising his finery.

Many political and journalistic reputations have been staked on taking this Prime Minister's abilities seriously. What people say next has to be squared with what they said then, so now they insist that Mr Brown's problem is a difficulty in “communicating” or “explaining” his vision; or that he has been sidetracked from “focusing” on his big ideas; or both. It's all there, they hint: it's just that we haven't seen it yet.

Speeches are made and columns written urging the wizard to hurry up and show us his magic. But the wizard hasn't got any magic. Poor wand-less Mr Brown isn't concealing or delaying his abracadabra moment. There's nothing there: nothing to get cracking with, nothing to communicate, nothing to explain.

I think his premiership is disintegrating. With no belief in the human at its centre I doubt the disintegration can be halted or reversed. I think this will become plain by autumn. One way or another, and very possibly before the next election, I think Mr Brown will go.

Times Recommends

• Strip away the figleaf and reveal naysayers
• Dave’n’George: there may be trouble ahead
• I do not accept that £64,000 a year for politicians is peanuts