As an opening line, “Am I alone in thinking...?” is a time-honoured trick of the hack-columnist's trade. But in the opinion that follows I will not be alone. There will be a small crowd of us. It is possible to be precise about its size. It is composed of virtually every sitting Member of Parliament, virtually every living former MP and most of the House of Lords. And almost nobody else. This opinion will pit - en bloc - our legislators past and present against the great majority of the rest of the public, probably including you, Times reader.

Ah well, here goes. I don't think the question of whether Mr Speaker Martin should resign is any business of mine, or yours, or the British media's, or the British public's. I think it's for sitting MPs, and for Mr Martin himself, to consider and decide. And in making that decision I doubt that he or they should take much notice of any of us.

I've surprised myself with this opinion. It is not what I expected to feel. I was still parliamentary sketchwriter of The Times when the present Speaker was elected. All of us sketchwriters thought him a pretty mediocre choice, and expressed ourselves with varying degrees of vehemence: a range in which I (who called him a drongo) came more or less in the middle. Another called him Gorbals Mick. Mr Martin and his friends were gravely offended by this cacophony of sneering commentators, and called us public-school snobs, or English racists.

In vain did we protest that many of us hadn't been to public schools; that had a dim-witted toff been dragged into the Speaker's Chair, we would have been just as quick to call him an upper-class twit or a chinless wonder; and that if a cockney Speaker called William had been elected we would have called him Barrow Boy Bill.

Anyway, in a free press in a free country we were entitled to our fun, and our views.

Mine (for what it's worth) modified a bit in the years that followed. I watched Mr Martin day after day from the Press Gallery, and in the end concluded that he was not particularly quick-witted, and often slow to stand up to the government front bench; but that he was adequate, if undistinguished, and by no means a disgrace to the Chair. As it happens, we'd had a run of three unusually good Speakers before him: the radio-friendly and charismatic George Thomas; the stoutly independent Bernard (“Jack”) Weatherill; and the adroit and adorable Betty Boothroyd with her indefinable aura of celebrity and (whisper this) shrewdly unadventurous approach to showdowns of any kind. Betty didn't want a fuss.

These three stood out among a long line of Speakers, most of whom could not honestly be described as better than a mixed bag. I do remember what it was like on the backmost of the back benches, waiting, and hoping, and waiting, and hoping, to be called. Commons Speakers do tend to be honoured, if not in the breach, then in mellow, misty recollection. The Chair, like Punch, isn't what it used to be, but then it never was.

Mr Martin was a good deal more competent than some whom living memory could still identify - and unlike some of them he at least stayed sober.

As we got used to his strong accent, we found in his presence in the Chamber - I did, anyway - a certain crumpled fatherliness, and kindly manner. He did stand up to Tony Blair on occasions, though perhaps less to Gordon Brown; and I don't recall any serious road accidents of Speaker's rulings, though there were some wobbles, usually corrected in the end after hurried consultations with the clerks.

Oh, come on, fellow journalists, Mr Speaker Martin may not have been a paragon among Speakers but nor was he a disgrace. The truth is he didn't turn into quite the disaster some expected. And until very recently he was chugging along more or less adequately, no doubt strenuously supported behind the scenes by his suffering clerks - but that is what clerks are for.

Then along came taxi-gate, Air Miles-gate, and “oh-my-God-maybe- he's-going-to-go-on-and-on” gate. These were compounded by Derek Conway-gate, and Martin's obvious disinclination to do much about the longstanding mess of MPs' allowances and expenses - bequeathed to him, let's be honest, by (among others) St George, St Jack and St Betty.

And it is at this point - and rather to my surprise - that the former MP in me pipes up. It's not that I support the status quo on MPs' expenses. I don't. It wasn't honest in my day and it still isn't. This is a swamp that needs to be drained, and the press and public are right to feel that the remittances we pay for the members we elect are our business as well as theirs; a national question; and now an urgent one.

For what it's worth I would double their salaries and abolish all their allowances apart from travel by public transport, leaving them to hire the staff they choose, and buy what second homes they please, out of their own pockets.

But where I cannot follow the British press is in any campaign to “get” Mr Speaker Martin. We are perfectly entitled to find out what we can about disbursements (and Sam Coates's blog in Times Online and his dispatches about the wonders of the MPs' food allowance, and Mrs Martin's taxis, have had me chortling with mischievous pleasure). But having laid out the information, press and public should recognise that we come up short of a line that I think we should not cross.

Mr Speaker is elected by MPs. He is their Speaker. He is there to champion their rights against overmighty governments in good times and bad - times when the public and its news media are, and times when they are not, on Parliament's side. Never assume the media will always back a Speaker against a popular PM. Margaret Thatcher's henchmen used the press against Jack Weatherill, but Jack wouldn't buckle to the Daily Mail.

A Speaker may make mistakes of judgment on the issue of what is really in parliamentarians' best long-term interests (and in my view, on the allowances question, this Speaker has), but MPs should be able to feel that he is answerable to them in the way he exercises that judgment, not to columnists, or sleuths, or pollsters with clipboards.

The stories of overmighty monarchs, overmighty executives, and overmighty prime ministers and chief whips are often told. But there is such a thing as an overmighty mass media too, and it would be a sad day when Commons Speakers felt under constant pressure to satisfy the press.

Let our elected representatives be the lightning rods to register and transmit public opinion. And let them be candid with their Speaker. But the public whose opinion a Commons Speaker should consult first, care for most, and even fear a bit, comprises a rather smaller electorate of 650-odd souls. And they do not include you, they do not include newspaper leader writers, and they do not - any longer - include me.

Times Recommends

• Strip away the figleaf and reveal naysayers
• Dave’n’George: there may be trouble ahead
• I do not accept that £64,000 a year for politicians is peanuts