It is often true that a prime minister’s greatest strength turns over time into his or her biggest weakness.

Gordon Brown is a masterly politician, expert at creating dividing lines, a brilliant tactician who got to Downing Street by managing different sections of the Labour Party into supporting him. But now it is his political brain that is his undoing — because he cannot fix the electorate.

Tony Blair’s refusal to fit in with Labour Party rules, his “hi, guys” informality, was attractive in 1997. But ten years later the failure to follow proper procedures had sucked him into the loans-for-peerages affair and created the “sofa Government” that allowed a sexed-up dossier to be used as justification for the Iraq war.

For Margaret Thatcher it was the housewife’s appeal. The grocer’s daughter, never without her handbag, made clear that she knew what it was like to struggle with bills and set out to create a property-owning democracy through council house sales. But her failure to understand the anger of homeowners about the poll tax brought her down.

John Major slipped through the middle to become Tory leader because he was the least offensive candidate — then turned into a weak, grey figure who struggled to keep his warring party under control.

So if David Cameron becomes prime minister, what is the asset that could turn into a liability? There is remarkable agreement among Shadow Cabinet ministers about the answer to this question. The Tory leader has a close-knit circle of friends and advisers — the danger is that this tips over into what is perceived as a privileged clique.

The political and social lives of the Notting Hill Tories are more closely entwined than any other Westminster set I can think of, either current or historical. They eat at each other’s houses, are godparents to each other’s children, share school runs, co-edit church magazines, holiday together — all between discussing the next stage in the politics of the post-bureaucratic age. This isn’t The West Wing, it’s an episode of Friends.

The backdrop may be London W11 but it could be a small, rather incestuous country village. They buy curtain fabric from Osborne & Little (owned by the Shadow Chancellor’s family) and diaries from Smythson’s (where Samantha Cameron is the creative director). When the MPs’ expenses were published it became clear that the Cameroons’ furniture store of choice was Oka — run by Viscountess Astor, the Conservative leader’s mother-in-law.

If the Tories win it will be Oka sofa government. “It’s dinner party politics,” says one Tory frontbencher. “There’s been a coup of chums.”

Many members of the inner circle have known Mr Cameron for years. There are several Etonians — including Ed Llewelyn, the chief of staff, and Oliver Letwin, his head of policy. George Osborne was in the Bullingdon Club with him and Kate Fall, his “gatekeeper”, read PPE at Oxford too; Andrew Feldman, the chief executive, met Dave on the Brasenose College ball committee, Steve Hilton, Mr Cameron’s chief strategist, worked with him in the Conservative Research Department.

Some new members have been admitted to the inner circle — William Hague is the link to the party grassroots, Michael Gove the intellectual ballast, Andy Coulson gets the message across to the tabloids — Mr Cameron’s loyalty to the former News of the World editor over recent allegations about bugging show how deep the bonds go.

The point is the Cameroons are not just colleagues, they are friends. It is telling that they want to re-create the informal network of offices they have in the Commons if they go into government. In particular, Mr Osborne and Mr Cameron are determined to have greater fluidity between No 10 and the Treasury.

Policymaking and strategy are driven by the gang. According to Tim Montgomerie, of Conservative Home, “the biggest decisions are drafted in very small groups”. One senior Tory claims that Mr Cameron has never had a group meeting with any of the frontbench departmental teams.

There are, of course, benefits to this approach. The Conservative high command is united and can react quickly to events. But there is growing resentment in the wider party about the exclusivity of the inner circle. It doesn’t help that its members are wealthier than most MPs, who are feeling the pinch after paying back some of their expenses.

Tory backbenchers are remarkably grumpy given the state of the polls and last week’s by-election win. Shadow Cabinet ministers complain about the lack of teamwork, the absence of political discussion and the failure to consult before decisions are made. It is said that Mr Cameron is so used to having his friends around him that he will not listen to those who tell him something he does not want to hear. “The cliquiness is awful,” says one frontbencher. “Everyone should be very excited about the prospect of power but they’re not. People are very demoralised, there’s not much enthusiasm because the Cameron lot are so disdainful of everybody else.”

Another senior Tory says that Mr Cameron is right to say he is the heir to Blair: “It’s sofa opposition. Let’s hope he learns the lessons from Blair of how dangerous sofa government can be.”

If the Conservatives win power they will be plunged straight into trying to fill a truly terrible hole in the public finances. Mr Cameron will need all the support he can get from his front bench as he tries to reach agreement on tough spending settlements. Assuming the Lisbon treaty has been ratified, he may also have to face down backbenchers still pushing for a referendum.

While he is an opposition leader ahead in the polls, Mr Cameron can afford to rely on a small clique, but if he becomes prime minister in such difficult circumstances, he will need a far bigger circle of friends. “Cameron has the potential to be a really significant prime minister,” says one frontbencher, “but he risks destroying himself because he won’t widen his circle.”

Times Recommends

• Bank charges hit us for being human
• Sorry, Father, thou still shalt not steal
• The shed where Santa’s failures go to die