Not all uplifting stories have a happy ending. When the Berlin Wall fell in 1989, to be followed within a couple of years by the collapse of communism in its Soviet heartland, it was as if history was intent on imitating JRR Tolkien’s fantasy epic, The Lord of the Rings. The Dark Tower in Moscow had fallen; the evil empire was over. The West had won.

The Russians deserved better times, too, after weary decades of Soviet tyranny but it did not come in the way we imagined. The new unfree Russia led by Vladimir Putin has some of the intransigence of the nasty old Soviet Union: it reacts to the humiliation of losing the cold war with belligerence.

This poses a real challenge for the West’s present crop of leaders who have no experience of great power politics. Russia or China – where our prime minister is seeing the sights this weekend – seem immutable. The temptation for Gordon Brown and his colleagues is to say that the business of the West is business and to forget human rights and forget our allies on the front line of democracy. That temptation must be firmly resisted.

For a few years in the 1990s I observed in Yeltsin’s Russia the profound shock that followed communism. Living standards were collapsing as the planned economy fell apart. Every adult over the age of 45 seemed without a future. In a country without law or title, gangsters, black marketeers and ruthless former party men prospered and the devil took the hindmost.

In a little town on the road to St Petersburg I went to the local supermarket. The huge window frontage was empty except for a few pitiful lumps of cheese and cans of condensed milk. The manageress explained the shortages: “The Jews have taken all the food. That Jew Yeltsin in the Kremlin has taken it all and sold the country to the foreigners.” In vain I tried to explain to her that Yeltsin was not Jewish. “Ah, but his wife is,” she replied triumphantly – and again, falsely. Like Germany after 1918, a defeated country drew on conspiracy theories and fear of foreigners to explain defeat.

Paranoia about the outside world suits Russia’s rulers, even though they personally know better. They tell their people that Britain and the West have designs on the Motherland, even as they send their children to British public schools and universities. Paranoia about foreign plots gives them an excuse to hinder contact between ordinary Russians and foreigners who carry the contagion of democracy.

Poverty and a breakdown of law and order have made it easy for Putin to push aside Yeltsin’s old cronies and take charge. The Communist party may have had its guts ripped out but one formidable institution of the old order remains intact: Putin’s old employer the KGB – or FSB, as that organisation is now called.

The Russian government was last week accusing the British Council of being “a nest of spies”. Paradoxically, the Russian government is itself a nest of spies. President Putin, soon to morph into a prime minister after more fixed elections in March, is but the most prominent former spy. The two most powerful people under him, Sergei Ivanov, the deputy prime minister, and Igor Sechin, the deputy chief of staff, are from the security services. Russia’s largest oil company, its largest shipbuilding company, Aeroflot, the airline, and the railway network are dominated by spooks. The management of the vast and ubiquitous Gazprom, with its holdings in TV, publishing and banking as well as energy, is also dominated by spooks. Russia is not a dictatorship, not a conventional autocracy: it’s a spookocracy.

“The state within the state has become the state,” argues The Economist’s Edward Lucas, author of a forthcoming study, The New Cold War: How the Kremlin Menaces both Russia and the West.

Under Stalin, chiefs of secret police were executed to stop them becoming overmighty. After the Soviet dictator’s death his politburo quickly eliminated his hated last spymaster, Lavrenty Beria, fearing his ambition. Yuri Andropov, a former KGB leader, briefly became Soviet leader in the early 1980s as the state began to crumble. His programme: reform the moribund economy but hold onto power at all costs. Andropov died before he could realise his dream. In 1999 Putin unveiled a plaque to him at the Lubyanka – the offices, prison and execution site of the KGB. He was “an outstanding political figure”, Putin said.

That is not to say that Putin is not genuinely popular with many Russians. Life has got better under his regime. The rise in oil, gas and world commodity prices has brought a boom to a resource-rich country. In this climate, the loss of political freedoms and the murder of a few nosy journalists seem bearable. This devil’s pact is well understood by Brown’s Chinese hosts, who still keep the title communist despite presiding over turbo-capi-talism and 10% annual growth.

Pessimists like Lucas think that Russia’s leaders are unimaginably cynical. He suspects that a series of appalling terrorist bombings of apartment blocks involving the deaths of hundreds of innocent Russians “were a ruthlessly planned stunt staged to ensure that Mr Putin would become the country’s undisputed leader”.

Whatever the truth of this murky affair, the bombings certainly were blamed on Chechen secessionists. Chechenya and its capital Grozny were shelled to pieces by the Russian army and Putin surfed a wave of outraged Russian nationalist fervour to popularity and power.

So when London granted asylum to Boris Berezovsky, Putin’s greatest oligarch opponent, and to Akhmed Zakayev, the Chechen resistance leader, along with more than 20 others, the stage was set for confrontation with Britain. London, like Paris in the 1920s and 1930s, has become a hub for Russians abroad. Dissidents mix with billionaires on spending sprees from Moscow, businessmen rub shoulders with secret agents. Plots and counterplots are hatched.

The United States is perhaps too powerful for Russia to confront directly; Germany under Gerhard Schröder, its former leader, bent over backwards to do deals over gas, while President Chirac’s France was a wayward ally of Washington that could be useful to Moscow. But Britain is an old imperial foe and MI6, its foreign secret service, although smaller than the vast CIA, is nevertheless respected.

A Berezovsky client, Alexander Litvinenko, a former FSB man, was marked out for assassination. If the plot to poison him with radioactive polonium-210 had gone undetected it would have been a great coup for Russia’s intelligence services. But it was botched. The consequent publicity has rebounded on the Russian government. It recalls the worst excesses of the cold war: poison umbrella attacks on dissidents and all the other bizarre paraphernalia of assassination.

By the law of unintended consequences, Russia’s hardmen are at last revealing their true selves to peaceable folk in Europe. After cutting off gas to Ukraine, who would now rely on Moscow as sole supplier for its energy needs? Germany and France, under new and more realistic leadership, are wary of Putin. However late in the day, the European Union is beginning to learn that it must be more supportive of Russia’s democratic neighbours.

What to do now the British Council and its St Petersburg boss are being harassed in time-honoured KGB fashion? Banning the Bolshoi will not do much good but we are right to stand up for the British Council, which does good work to ensure that ordinary Russians have access to information about the West. We should make a stink about it in every international forum.

The West cannot always win against a hostile power in the short term as the cold war proved. Out of self-respect, if nothing else, our leaders need to show support for people at the sharp end of autocracy. We need to speak up for harassed liberals in Russia and, through bodies such as the Westminster Foundation, openly promote democracy. We need to back democratic friends in adjoining Georgia, Ukraine and Belarus. For Brown to give up on them now would be worse than a mistake. It would show an utter failure of nerve.

Times Recommends

• Chelsea Flower Show has lost the plot
• The age of personal vitriolic abuse
• Obama: the new Great Redeemer