Wednesday night. Cipriani, off Berkeley Square. A quiet family party at the back are speaking heaven-knows-what and eating Italian comfort food at £65 a head. A posse of hungry-looking money men in suits swallow champagne before being shown to their table. They are distracted (as am I) by two beautiful women sinking bellinis at the bar. And through the revolving door comes a man with a name like Davos. He has a furry torso busting out of his blazer, and a Blancpain watch the size of a beer mat.

Federico, the maître d’, goes through an elaborate pretence of not knowing who Davos is, then yells at the white-jacketed staff to fix him his usual apéritif and seat him at table 28.

Are these the creatures I have been looking for? The favoured ones, whom fate has blessed with money and the taxman with the mother of all loopholes? Are these the people who have made London the most cosmopolitan and expensive city in the world; the steel-smelting, penthouse-collecting Brahmins of 21st-century Eurasia? The people who, though they pay little UK tax, may contribute as much as £5 billion to the economy in other ways? Are these the wildly rich, non-domiciled UK residents known to their accountants as non-doms?

One of Federico’s colleagues has rashly confirmed as much over the phone. Another close observer of the scene has identified this place, along with Annabel’s and the George nightclub, as prime non-dom party locales. But they are safe here. Federico will only shrug. “This is London,” he says. “They come from everywhere.”

Of course they do. But London should not kid itself. It’s not Gordon Ramsay who secures for W1 the continued patronage of Roman Abramovich and Boris Berezovsky. It’s not the Royal Ballet (as far as we know) that induced one Qatari to pay £100 million for an as-yet unbuilt flat on Hyde Park. And it’s certainly not the weather that attracts the foreigners who comprise roughly half the hedge-fund managers in London, the hedge-fund capital of the world.

Above all, it’s the tax system, which takes 40 per cent of the average middle-class family’s earnings but only token amounts off non-doms, and then only if they are honest. This is the system that, in April, led the IMF to bracket London with Bermuda as an “offshore financial centre” (translation: tax haven).

Yet it is also a system that, for all its apparent iniquities, has put the international ultra-rich in the service of UK plc to the extent that we may no longer be able to manage without them. As one sage has put it, for Britain to tinker with its non-dom laws would be like Saudi Arabia giving up its oil.

Even so, tinkering is what both main political parties plan to do, with incalculable consequences.

On Monday, George Osborne thrilled the Conservatives with his promise of an end to inheritance tax as we know it, and he promised to fund this promise with a £25,000-a-year levy for the privilege of non-dom status. He claimed that this could raise up to £3.5 billion.

Osborne’s pledge has triggered a furious political row over non-dom numbers. Broadly defined, non-doms include all foreigners in Britain. No one knows how many there are, let alone how many would find it worthwhile to pay the levy rather than pay UK tax on all their worldwide income. The Shadow Chancellor put the figure at 150,000. Labour hit back with 15,000 – but the Treasury has since disowned the smaller figure, and so has much of the high-end accountancy profession.

“Labour’s numbers are nonsensical,” says Mike Warburton, of the City firm Grant Thornton. On the basis that the numbers filing tax returns as non-doms nearly doubled to 112,000 between 2003 and 2005, he finds an estimate of 150,000 to 200,000 for 2007 “entirely reasonable”. More to the point, Warburton believes that at least 120,000 of these will have enough foreign assets to want to protect them by paying the levy rather than UK tax.

One-nil to the Tories, then. But what do the non-doms themselves make of paying £25,000 a year for a status that they have enjoyed free of charge for decades?

And who are they?

They fall into three broad categories. By far the most conspicuous are the über-rich, who work only if they want to, whose assets are genuinely global, and who regard an Osborne-style levy as small change.

Frederick, a Belgian banker, is non-dom-eligible by virtue of his passport. It’s as simple as that, for as long as he chooses to live in London: “My advisers have always told me to just tick the non-dom box [on my tax return],” he says. Then his non-UK income and capital gains need not be taxed by anyone.

Speaking carefully, in the third person, Frederick says of the £25,000 levy: “For people with [foreign] assets, it doesn’t matter. It’s nothing, not important.”

Chump change, in other words – and this is the overwhelming consensus of the seriously rich and those who minister to them. To the non-doms who walk into Nico Valdes-Scott’s Chelsea branch of Strutt & Parker in search of London accommodation, £25,000 is “like a couple of quid”. To those who read Spears Wealth Management Survey, edited by William Cash, it’s “not even the service charge in a decent apartment block near Hyde Park”.

The tip of the non-dom iceberg will clearly not be alarmed by George Osborne’s proposals. In fact, they would be welcomed for bringing clarity where doubt and resentment is on the rise. This goes for London’s 23 foreign billionaires, led in the brute wealth stakes by Lakshmi Mittal, the Indian-born steel tycoon, and Roman Abramovich. It goes for a fast-growing cohort of Indian industrialists; for the Arab princes who have long used Knightsbridge as a home from home; for the Kazakh and Ukrainian oligarchs following their Russian brethren to Belgravia; for the 300 foreign-born footballers now in the Premiership (up from just 11 in the First Division in 1992 – their earnings for all games played abroad are tax-exempt); and for senior foreign-born bankers running City operations that are likely to remain in London for its language, time zone and communications as well as its tax regime.

This is the global plutocracy for which London has become an unrivalled service centre. Its members are not troubled by Tube strikes; they glide around in chauffeur-driven Bentleys and Range Rovers. They bypass Heathrow’s queues in favour of Farnborough. Their businesses create thousands of high-paying jobs (the London-based Greek shipping community is estimated to provide 4,500 alone). Their drive and flair accounts for much of the City’s dynamism.

But they represent only a fraction of the non-doms. The second category consists of the merely wealthy, most of them foreign-born but City-based. Many are in London on three to five-year investment banking assignments made more attractive by non-dom status because it lets them park their savings offshore to appreciate tax-free. The Tory plan could hit them hard: with £1.2 million or less abroad, they would be better off paying UK tax on their worldwide income and capital gains – or moving elsewhere.

The third and largest category is even more likely to feel squeezed by a levy. It consists of thrifty, well-paid professionals who happen to have strong foreign ties – “an Australian engineer with significant savings back home who happens to be working in Milton Keynes, for instance,” says Patrick Stevens of Ernst & Young. How many fit this profile? No one knows.

Hit the streets around Sloane Square and the complexity of non-doms’ lives begins to emerge. By no means all of them are sanguine about parting with £25,000 a year.

“Why should I pay that?” asks Nina, 31, who runs cosmetics businesses across Europe. “I’m domiciled in Slovenia but I can operate in any country. If they decided to tax me, I would move to Monaco.”

Eric, a 39-year-old French strategy consultant who starts a new job in the City on Monday, has not heard about the Osborne levy, nor about Labour plans to crack down on abuse of non-dom status. “That’s crazy,” he says, when I explain that he would have to choose between the levy and paying 40 per cent tax on his French savings. “You come to London to make money, not for the lifestyle. This tax stuff would really weaken the Anglo-Saxon model.”

Annie, a London mother of three, is not a non-dom – but her Lebanese husband is. “This wouldn’t make us go back to Lebanon,” she reflects. “But it might make us think hard about voting Tory.”

It was a variation on this sort of threat that caused Margaret Thatcher a rare loss of nerve in 1988. With her Treasury minister, Norman Lamont, the Iron Lady was considering taxing non-doms on their worldwide income – a far more stringent crackdown than a levy. Legend has it that she received a call from a representative of the Greek shipping community who let her know that they would up sticks en masse and move to Piraeus if she persisted; and that she swiftly dropped the plan. “All true,” Lord Lamont of Lerwick told The Times this week.

It may not be surprising, then, that Gordon Brown’s pledge in 1994 to close the non-dom loophole should Labour come to power was quietly forgotten after 1997. It has since been revived, thanks partly to embarrassments over non-dom Labour donors and the IMF’s designation of London as a de facto tax haven. But even talking tough on non-doms remains risky. Hedge-fund managers have only to reply that they might move to Zurich, and curbing non-dom privileges begins to look like a net loss to the Exchequer. “And the Monaco property market is booming,” a partner at Knight Frank muses. “These people already feel on safer ground there.”

Who can be a non-dom?

To be eligible for non-dom tax status you must be foreign-born or born to a father who was foreign-domiciled – which does not have to mean resident – at the time of your birth.

To apply for non-dom status could not be simpler: just tick the box on your tax return. Any arguments with HM Customs and Revenue come later.

New applicants may have to demonstrate regular travel to their homeland and an intention to return there permanently at some point. Most importantly, any income earned abroad must stay abroad to be tax-free. The system has long been used for tax avoidance. Methods include setting up “split” contracts - one for fully-taxed UK income, the other for tax-free “nonUK” earnings. Money brought into the UK as capital rather than income can also be tax-free.

“The really big carrot” for non-doms, according to Caroline Garnham, a lawyer with LG-Legal, is tax-free offshore capital growth. This would include that of lavish London properties bought by offshore trusts.

For one group of foreigners in London, nondom status means nothing: Americans are taxed in the US on their worldwide income wherever they reside.