What should we make of the failure of the Home Office to operate control orders properly, the MTAS computer fiasco at the NHS, and Ruth Kelly’s climbdown on home information packs?

Almost everyone has one of two responses. Some say that these are isolated failures in an otherwise acceptable record, others that they are evidence of a general incompetence that has a simple solution – to put different backsides on Cabinet chairs.

Almost everyone is wrong. There’s a third possible reaction: that these episodes (and there are countless others) show that centralised hierarchy is a terrible way of getting things done. Policy failures aren’t due to having the wrong personnel in charge. Nor are they exceptions to the rule of general competence. They are the inevitable result of bad organisational structure.

There are four lines of thinking that tell us this. One comes from Friedrich Hayek. Knowledge, he said, is inherently dispersed. No single minister or boss can know very much, so decentralisation is necessary to maximise our use of knowledge. The collapse of the Soviet Union proved him right. This raises the question: if a centrally planned economy is a stinking idea, why is a centrally planned health service or educational system a good one?

The second line is the research into “cognitive biases” inspired by Daniel Kahneman, of Princeton University, and the late Amos Tversky. This has given us a long list of errors of judgment that even experts make. It’s been fashionable for economists to show that investors committed these errors when choosing and buying shares. But if investors can be systematically irrational, why should ministers and civil servants be any more rational, when they lack the investors’ advantages of big incentives, clear and immediate feedback and (we might add) high intelligence?

The third line is “transactions-cost economics”. Inspired by the Nobel prize-winning economist Ronald Coase and developed by Oliver Williamson, of the University of California at Berkeley, this compares the costs of doing things within the marketplace – the difficulties of assessing quality or of writing contracts – with the cost of doing them internally.

This research has revealed that the costs of internal organisation can be big. Staffan Canback, of the management consultants Canback Dangel, estimates that the operating performance of manufacturing firms deteriorates when it grows beyond quite a small size. And if this is true in manufacturing firms, how much more likely are they to matter in bigger organisations such as government departments?

One reasons why big organisations become inefficient is communication failure. Subordinates have lots of reasons not to tell bosses the truth. They don’t want to burden “busy” people with detail, or rock the boat, or be victim of “shoot the messenger” syndrome. The upshot of this was famously described by the late Kenneth Boulding: “The larger and more authoritarian the organisation, the better the chance that its top decision-makers will be operating in purely imaginary worlds.” Those photoshopped pictures of Patricia Hewitt dressed as Comical Ali tell a truth: that government has learnt little in the 40 years since Boulding wrote those words.

The fourth strand of thinking is the recent management literature, such as The Myth of Leadership by Jeffery Nielsen and The End of Management by Kenneth Cloke and Joan Goldsmith, arguing how counter-productive “leadership” can be. Hierarchy, they argue, stifles originality and creates an organisation with egomaniacs at the top and slovenly, dehumanised automatons at the bottom.

There is, then, a vast amount of economic thinking that enjoins us to be sceptical of large hierarchical organisations. For years, big businesses have been “delayering”. Sir Terry Leahy, who runs Tesco, claims there are only six layers of hierarchy between him and the woman on the checkout. How many layers are there between Patricia Hewitt and a nurse? Other businesses, such as Koch Industries in the US, are going farther, trying to replace internal hierarchy with “market-based management”, using internal markets rather than central command.

Customers too are recognising that hierarchy doesn’t deliver as well as more cooperative businesses. Verdict Research reports that the cooperatives John Lewis and Waitrose are Britain’s two favourite retailers. Waitrose’s sales are growing faster than those of other main food retailers. And in the City, hedge funds, most of which are partnerships, have been winning business away from more traditional, hierarchical fund managers.

It’s not just in the high street that hierarchy is being rejected. It’s also being rejected at the ballot box, with support for the hierarchical parties falling. Here, the political class is in denial. At the recent local elections in England, five out of six voters choose not to vote Conservative, mainly by not voting at all. And yet when David Cameron described this as a “stunning” result for his party, no one pointed out that he was talking rot.

What we see here is a pure ideology, as much so as any juvenile Marxism; I call it managerialism. The problem is not that the political class has considered the arguments against big hierarchical structures and rejected them. They haven’t even thought about them. There’s no evidence that the break-up of the Home Office was based in transactions-cost economics, even though this is how economists routinely think about mergers and demergers. And Mr Cameron said last week: “I don’t follow my party, I lead them”, as if leadership were a self-evidently good thing rather than macho posturing.

Then when voters show their opposition to all this by not voting, the political class moans about apathy, confusing apathy with contempt.

But there are alternatives to managerialist politics. We could have flat-rate allowances paid to everyone rather than an administration-heavy welfare state; schools and hospitals could become worker coops; we could use demand-revealing referendums rather than look to “leadership”. The question isn’t: “Are there alternatives to managerialist politics?” It’s: “Why do people fail to even consider them?”

Times Recommends

• Tear down this wall! And save the planet
• Double click if you want the bin emptied
• Have you got permission to read this?