I can see that it must be infuriating. The planning application went in two years ago, the plastic models were growing dusty, the Candy brothers had all sorts of wild ideas to provide fur fridges for the mink coats and heated tarmac in the car parks.

Chelsea Barracks was to become the new air-conditioned Belgravia, turning the property-developing Candys into the next Dukes of Westminster and allowing Lord Rogers of Riverside to compete with Sir Christopher Wren's Royal Hospital. Then Old Jug-Ears parped and the whole thing was called off. You'd feel irritated if you had spent years on a design and the Prince of Wales intervened at the last minute to scupper it.

Lord Rogers wants a national inquiry into Prince Charles's role in “single-handedly destroying” the £3billion development, saying that the Prince has broken “a bond of trust” with the British people. The peer of the realm says that it must never happen again, that the Black Prince must be muzzled and a committee of independent constitutional experts must vet his involvement not just in architecture but medicine, agriculture, the arts and the environment. It is an outrage for anyone unelected to have any power, he says. (He has obviously forgotten that, as a Labour peer, he also has unelected influence.)

Lord Rogers is magnificent in his fury - but wrong. There are plenty of issues that the Prince of Wales can't touch: Gordon Brown, the constitution, Europe, the Middle East and whether the country goes to war. Architecture is not one of them.

What are the royals for if not to protect our heritage? He's meddling, you say, but the Prince is at his best when he becomes involved. The neighbours never wanted these glass and steel high-tech residential towers stuffed with £50million flats. But they had no influence over the combined might of Lord Rogers, Gulf State money and the Candys (who have two other vast projects in the capital) and who are held in awe by London's planning committees.

Prince Charles could do something. He pulled rank and wrote a letter to his friend, Sheikh Hamad bin Khalifa al-Thani, the ruler of Qatar, who was backing the project. The arrogance of architects has been trumped by the arrogance of princes.

What do you want the heir to the throne to do? Do you want a vacuous, empty-headed playboy or a serious person (albeit one in a kilt and holding a crook) prepared to join in the argument about community versus ideological modernist architecture.

Over the past 40 years, the meddling Prince Charles has had an uncanny knack of championing causes decades before they became popular. The former Cabinet minister Charles Clarke once derided him as “old-fashioned and out of time” but he is curiously prescient for a man surrounded by flunkeys and courtiers.

He was still experimenting with cherry brandy when he first talked about the environment. He was derided for instigating a bottle bank at Buckingham Palace years before councils thought of recycling. David Cameron now espouses his call for more localism, Tesco is catching up with his views on organic food. Long before 9/11, he was talking about Islam and the need to understand the underlying religious tensions in this country. With his Prince's Trust he was years ahead of Sir Alan Sugar in encouraging young people to start their own businesses.

He has set up projects to embrace his beliefs about social deprivation, community cohesion, urban planning and Britishness, firing off letters to ministers suggesting the elderly may not be having a nice time in hospital.

But no one will be hanged, drawn or quartered if they don't agree, nor will his interfering in these arguments cause a constitutional crisis. What will ruin the Royal Family is being too flippant rather than being too involved. A recent survey by ICM asked: “How much do you think that the Prince should speak out on more political matters?” Nearly a third, 29 per cent, said more often and nearly half, 45 per cent, said about the same as he does at present.

What damages the monarchy are pictures of the young princes stumbling drunk out of Boujis nightclub at three in the morning, revelations that Prince Andrew takes a helicopter to a golf course or that Princess Eugenie costs the taxpayer £100,000 in police bodyguards as she flits around the world in her bikini.

When Prince Charles became Prince of Wales he was given no sense of purpose or direction and no idea what to do with his life. Unlike Edward VII, he was not prepared to fill the idle hours by going shooting, nor could he accept Harold Nicolson's advice to “bow his head obediently to this cruel fate”.

Instead, he has created a role for himself over the past 40 years as an activist. He has been at his worst when complaining about his gilded life while ordering too many boiled eggs, his best when concentrating on using his “privileged position” to question the grip of the architectural, medical and educational establishments.

The damage to the Royal Family comes when they are seen to be useless. So long as they are being useful and have a sense of purpose, the monarchy is safe.

Times Recommends

• Fundamentalist brew: faith and power
• Big guns wheeled out for the marginals
• Wanted: new merchants of light