The date is October 2005 and Tony Blair is addressing the Labour party conference in Brighton. He has just won another landslide election victory and the war in Iraq is a long time behind him. He is, understandably, in triumphal mood.

“Never was an election victory more justified. The government’s achievements speak for themselves. Our opponents are in terminal decline. We can expect to do even greater things in the years, the decades, to come.”

But then he pauses. There is a perceptible change of mood. The delegates tense. There have been whispers for many months of something big in this speech, something way beyond the normal party-boosting rhetoric. They are right.

“When I say ‘We’ let me be clear what I mean. I want to be frank with you. Brutally frank. The government has served the nation well and the voters have responded.

But you — the Labour party — have been a great disappointment to me. Too often you have failed to follow where I have led. Too often you have cavilled and questioned. Too often the government has done great things in spite of you and not because of you.

“The time has come to change. You’ll simply have to go. You’ve become a hindrance to good, strong government. Quite what will replace you I have yet to decide. I have set up a Downing Street taskforce led by Lord Alastair to look at the options, and you will be informed of its conclusions. Thank you and goodbye.”

Absurd, of course, though Mr Blair must occasionally dream of making such a speech — especially after a week when so many of his own MPs turned on him. But my little fantasy leads to a real question about the democratic basis on which modern prime ministers exercise power.

Democracy boils down to a choice between two options. In one we force the government to do what we want through endless referendums on all the big issues. Modern government is too complicated for that. Endless plebiscites would result in paralysis.

In the other we let them do what they want and kick them out if we don’t like it.

But we don’t vote for “them”. We vote for a party. And the party is the vital link between the electorate and those in power. That link begins to weaken if the leader does too many things that seem to be at odds with his or her own party. He derives his power from the party and his democratic mandate from the people who elected the party. If he is significantly out of step with his party, what is his relationship with the voters? Blair is hardly the first prime minister with a big majority to do things that have upset large sections of his party, but there does seem to be more of it about these days. The party’s manifesto said there would be no top-up fees for university students yet Downing Street wants them, so that is the new policy. The same manifesto said a reformed House of Lords would include a democratically elected element, but Blair now wants them all to be appointed.

He believes in market forces driving the NHS and the need for foundation hospitals. Many in his party, including his chancellor, do not. And then, of course, there is war with Iraq. Most MPs who spoke against it on Wednesday were reflecting the concerns of party members and constituents.

It is true that a prime minister does not need parliament to declare war. He has the royal prerogative tucked in his back pocket — a powerful vestige of entirely undemocratic monarchical power that allows him to go to war and do much else. And that reinforces the point. As the historian Peter Hennessy puts it, the office of prime minister is “Napoleonic”. We have a de facto presidency. And it looks as if it is here to stay. If we don’t do something about it there could be problems ahead.

The Labour MP Graham Allen, once a government whip, has written a book called The Last Prime Minister — Being Honest About the UK Presidency. Allen is not whining about our over-powerful prime minister. He thinks we should acknowledge the reality, welcome it and find ways to get it to operate more effectively and — crucially — more democratically.