CHARLES CLARKE has always respected Gordon Brown’s abilities as a minister and politician. But he has equally had doubts about his personality and leadership qualities.

Cabinet ministers who have known both for years say that they have never seen Mr Clarke kow-tow to Mr Brown, and he is virtually alone in that.

Where Mr Brown may have dominated, even frightened, some ministers with the sheer weight of his intellect, he has never had that impact on Mr Clarke. Indeed, in a savage twist, Mr Clarke suggested yesterday that part of Mr Brown’s problem was that he lacked confidence and was nervous.

Ministers who sat in the Cabinet with them regard them as intellectual equals, perhaps the two cleverest men in the Labour high command. They are the same age, 55. If necessary, Mr Clarke would stand up to Mr Brown, and there is obviously a grudging admiration between them despite the frostiness in their relationship.

No one could call them friends but there have been civil links between the two. Norwich City Football Club may have had a small part to play in what many saw as an improvement in their relations in recent years. Mr Clarke is one of the local MPs and an avid fan, as is Ed Balls, Mr Brown’s closest ally and now Economic Secretary to the Treasury, and they have often met at matches.

Mr Clarke is very much his own man and Downing Street found laughable yesterday suggestions that somehow it could have reined him in. It was Mr Clarke who only a few months ago was sacked by the Prime Minister and who turned down at least two offers from Mr Blair of alternative jobs.

Similarly, while most Cabinet ministers have swiftly fallen in behind Mr Brown, whom they expect to take over, no one really expected Mr Clarke to try to curry favour so that he could get back into the Cabinet. That is not his style.

Mr Clarke has not quite ruled out a leadership bid himself, although he has insisted that nothing he has said in recent interventions should be seen as part of a campaign.

The former Home Secretary’s doubts about some aspects of Mr Brown’s make-up go back a long way. Mr Clarke was the pugnacious chief of staff for Neil Kinnock for nine years. During much of that time, Mr Brown was No 2 in John Smith’s Treasury team. In a foretaste of what Mr Brown was later to do under Mr Blair, Mr Smith ran the Shadow Treasury team often independently from the leader’s office, leading to regular clashes, and an astonishing attempt by Mr Kinnock to impose a change in the policy on taxation.

Relations between Mr Clarke and Mr Brown were said at the time to be frequently uneasy. When Mr Smith became leader, there was no reconciliation and for a short time Mr Clarke was in the wilderness.

Things were not that much better after Mr Clarke entered Parliament in 1997. Mr Clarke was a strong enthusiast for entry to the euro and was angry at the way Mr Brown quashed any chance of it so early in the Labour years.

So Mr Clarke was not holding back when the London Evening Standard got to him on Thursday. Mr Brown should have reined in the backbench rebels who called for the Prime Minister to quit.“What he should have done was come out strongly and distance himself from them,” Mr Clarke said. “He could have done that with a click of his fingers. This has been complete madness.”

Of Mr Brown’s Downing Street grin, Mr Clarke said: “A lot of people are very upset and cross about that. It was absolutely stupid, a stupid, stupid thing to do.”

Mr Clarke said that Mr Brown must “prove his fitness" to be Prime Minister. “Part of the problem is that he lacks confidence. He is nervous,” he said.

“That could all change when the burden of waiting for the job is lifted from his shoulders, and I think it probably will. But the problem is, nobody really knows. He is not where he should be at the moment. He is talented and brilliant but there are these little incidences like the grin in the car that build up a terrible picture."

For good measure Mr Clarke said that Alan Milburn, the former Health Secretary, was “leadership material”.

Yet it would not be a total surprise if Mr Brown brought Mr Clarke back into the Government. He would admit that a Cabinet that did not contain Mr Clarke would not be the strongest he could field. Yesterday showed how dangerous Mr Clarke can be if he is left in the stands too long.

Times Recommends

• Time-worn Freddie gives up Test matches
• Brighton company 'involved in cyber attack'
• The art of survival: it’s all in the mind