One must be empathic. If I was a rank-and-file reactionary Conservative, forced to swallow political failure for more than a decade, and now permitted, lizard-like, to come out of my smelly culvert to claim a place on the sunny rock, I might let the light go to my head too. I might preen my scales and tell tales of the decline - no, the breaking - of Britain under Labour.

But one can take empathy too far. It seems impossible to counter the triumphal gloominess of the old Right with anything as feeble, as unconvincing, as facts. The best figures available show crime has gone down, but we know, we know, we know it has gone up! The best figures available suggest improving performance at GCSE and A levels, but we know, we know, we know that this is because of a dilution in standards!

Then along come the Olympics, and the national narrative, for a moment, no longer favours the lizard class and its story of decline. So let me make the most of it, in this short interval before pessimism sets in again. In 1996, after 17 years of Conservative government, the past six under the premiership of the cricketing Major, Great Britain went to the Atlanta Olympics and won precisely one gold medal. We ended that Games in 36th position, just behind Ethiopia and just ahead of Belarus. It wasn't just that Greece did better than us - Kazakhstan got four golds. If one were to take the Olympics as any kind of indicator of national health (and why should we not?) we would have to conclude that the past 12 years have been very well spent. And if Gordon Brown is to get it in the neck for every ill, real and imagined, why should he not get some credit for this?

Of course, I know it's not as simple as that, and I can acknowledge that the National Lottery, set up by John Major in 1994, is likely to have been a big factor in our changed sporting fortunes. But however we divide up the accolades (other than to the sporting men and women themselves), what seems clear, to me and to Boris Johnson, is that this success hardly points to our living in some kind of brutalised, boneless pre-dystopia.

A moment here to savour the absurdities of political language. Last week the Mayor of London, surfing the new narrative, praised modern British youth, adding: “If you believe the politicians, we have a broken society, in which the courage and morals of young people have been sapped by welfarism and political correctness. And if you look at what is happening at the Beijing Olympics, you can see what piffle that is.”

This credo was written, not spoken off the cuff, and must therefore have been the product of more than the usual amount of Johnsonian carefulness. And as we all know (and Boris knows it too) the main politician complaining of the broken society is not Gordon Brown, is not David Miliband, and nor is it yet George Osborne. It is the leader of the Boris's own party and the most likely bet for the next prime minister, David Cameron.

In an age less forgiving to Conservatives and more serious about its senior mayors, much might have been made of this pretty central disagreement. After all, either British society is broken or it isn't. And the way that the Conservative Party dealt with this disagreement was characteristically good-humoured, tolerant and ridiculous. “Boris is Boris,” a spokesthing was quoted as saying. “He has to do what's right for London, while we are looking at the national picture. They won't always meet in the middle.”

What could this mean? Did it mean that society was broken everywhere except the capital (they're buggered in Bognor, but peachy in Peckham)? Or that it was broken throughout Britain, but that it was somehow right for London - being a sensitive kind of a place - to pretend that it wasn't broken there? Either way, it was nonsense. Boris's own retrospective gloss was to praise his leader for highlighting “serious and destructive social breakdown”, while expressing a determination to allow every teenager to reach his or her own potential, like the Olympic athletes did. There's lacking in contrition for you.

The analysis of Britain as being a broken society manages simultaneously to be wrong, irritating and - worst of all - suggestive of a whole series of wrong and irritating policies to come. Britain is demonstrably less “broken” than it was in the late 70s and at the height of the Thatcher era in the mid and late 80s. When Colin Moynihan was Mrs T's Minister of Sport the great issue confronting him was how to take on new police powers to deal with soccer hooliganism. Now Lord Moynihan is the chairman of the hugely successful British Olympic Association.

If the “broken society” only means that there are places where there is too much poverty and crime, and that any death caused by a knife or a gun is a tragedy then, this side of Paradise, it means nothing. This is the peculiarly irritating aspect of the phrase. To take just one side of modern Britain with which I am familiar now that my drinking days are over, hundreds of thousands of Britons are involved, as participants or supporters, in scores of sporting events: marathons, half-marathons, ten-kilometre runs, bikeathons, triathlons, duathlons, from Orkney to the Isle of Wight. Some are athletes, some are motivated by charity, some - like me - are recovering lard-arses. Are they part of a broken society?

I was struck this week by Adam Sage's story in these pages about the French families who are sending their children to learn English by staying with British families in France. They speak English in the home and then go horse-riding in the safety of France. “I don't want to say bad things about Britain, but you do hear horror stories about children sent to stay with families there,” remarked one French teacher. But I would be prepared to bet that what really motivates such Gallic fearfulness is media coverage of the supposed brokenness of Britain.

Fine. How many times have we been told that there are two Olympic-size swimming pools in the whole of Britain and 29 (or something improbable) in Paris alone? So how come we came third in the swimming medals and the French came ninth? Are they 20 times as broken as we are?

The lizards may get their new government. If so it should begin its rule by admitting what its predecessors - the party of 2012 - got right.

Times Recommends

• Fantasy world fuelled by sex, drink and drugs
• In short, the story is growing on us again
• A literary feast that I can’t wolf down