Two hundred years ago this month, in the middle of the Great Indian Desert that separated British India from the uncharted lands to the northwest, British soldiers encountered Afghan warriors for the very first time.

The British force, led by a Scottish diplomat with the splendidly imperial name of Mountstuart Elphinstone, consisted of several hundred near-mutinous sepoy (native Indian) troops, a handful of white officers, 600 camels, and a dozen elephants loaded with gifts.

Elphinstone, the first European diplomatic envoy sent to Afghanistan, had been dispatched from Delhi to coax the “King of Caubul” into an alliance against Napoleon, to explore this terra incognita, and - in the unlikely event that he survived - to report back to London on the “wild and strange” land beyond the mountains.

For a month Elphinstone slogged through the desert wastes, encountering bandits, warring clans and ferocious tribal chiefs off their heads on opium and alcohol who could be spoken to only in the early afternoon, that being the “interval between sobriety and absolute stupefaction”. For guidance, he had to rely on accounts of Alexander the Great's expedition, written more than 2,000 years earlier.

Just inside the border of what is now Pakistan, on November 21, 1808, Elphinstone was met by a body of 150 Afghan mounted troops, riding two to a camel, terrifying bearded figures each carrying a glittering matchlock musket. “Their appearance,” he recorded with fine Scots understatement, “was altogether novel and striking.” No one in his party could understand a word that they were saying.

So began the first formal contact between the British crown and the fractured state of Afghanistan. The latest chapter in that story will be written today, when the Afghan President, Hamid Karzai, arrives in Britain for the Prince of Wales's birthday party.

Astonishingly little has changed in the intervening two centuries: the Afghans are still regarded by Britain with a strange mixture of awe and incomprehension. The histories of our two countries are entwined like no other, yet that history has been routinely and tragically ignored or forgotten.

At 29 years old, Mountstuart Elphinstone was absurdly young, entirely fearless and very slightly mad. But he was also highly intelligent, fluent in Persian and Hindi, and profoundly observant. His book about his Afghan adventures - An Account of the Kingdom of Caubul - remains one of the most perceptive surveys of Afghanistan yet written, with such sublime chapter headings as “Rapine - how occasioned”.

In many ways, the society he described has barely changed.

The first meeting between the British diplomat and the Afghan King took place on February 25, 1809, in Peshawar, east of the Khyber Pass. As David Loyn recounts in Butcher and Bolt, an excellent new study of British engagement in Afghanistan, Elphinstone was stunned by the opulence of the Afghan king's outfit - “one blaze of jewels” - and the huge diamond, the cursed Koh-i-Noor, hanging from his wrist.

The British visitor was also intrigued by the Afghans' strenuous fitness regime, and one exercise in particular. “The performer places himself on his hands and toes, with his arms stiff and his body horizontal... he then throws his body forward, and at the same time bends his arms, so that his chest and belly almost sweep the ground.” This special form of exercise torture has been inflicted on soldiers and schoolchildren ever since. Elphinstone had discovered the Afghan press-up.

But he also noted something more profound. For all the finery, Afghanistan was chronically unstable, and about to tear itself apart. He compared it to ancient Scotland, a place of complex tribal animosities, clansmen raised on mountain hardship and bloodshed, riven by untraceable feuds and alliances, where central authority barely extended beyond the royal palace gates.

As so many British observers have been, he was drawn to the Pashtuns, with their fierce code of honour. “Their vices are revenge, envy, avarice, rapacity and obstinacy; on the other hand, they are fond of liberty, faithful to their friends, kind to their dependants, hospitable, brave, hardy, frugal, laborious and prudent.” They spent most of their time fighting each other, but they swiftly combined to repel any outsider and were ever “ready to defend their rugged country against a tyrant”.

Afghanistan, he wrote, could be understood only through its kaleidoscopic tribal structures. “The societies into which the nation is divided possess within themselves a principle of repulsion and disunion too strong to be overcome,” he noted. When the British marched into Afghanistan to bring about regime-change a few years later, Elphinstone, now in retirement, advised that the venture was hopeless.

Sure enough, in 1842, 16,000 British soldiers and camp-followers were slaughtered during the retreat from Kabul, the worst military disaster the Raj had suffered.

Britain's subsequent misadventures in Afghanistan followed a similar pattern, as successive generations of diplomats and soldiers sought to shape policy and impose change from outside without regard to local circumstances. In 2001 the EU representative to Kabul, Francesc Vendrell, candidly admitted that, since democracy had apparently been implanted in the country, “it was not thought necessary for us to understand the tribal system”.

There are small signs that the realities Elphinstone understood two centuries ago may finally be sinking in: General David Petraeus, fresh from Iraq, plans to enlist tribal leaders against the Taleban; a political settlement, John Hutton, the Defence Secretary, insists, is essential to a long-term peace.

Barack Obama has pledged greater commitment to the Afghan conflict, and will soon be making his own perilous journey into the diplomatic and military wilds of Afghanistan. Before he sets off, he might ponder the insights of the very first emissary to that beautiful and benighted place, and the words of an Afghan tribal elder who accosted Mountstuart Elphinstone, two centuries ago, to explain his turbulent world.

“We are content with discord, we are content with alarms, we are content with blood,” the old man said. “But we will never be content with a master.”

Times Recommends

• Five golden rules for regulating banks
• Uighurs’ cry echoes round the world
• It is not just democracy that is illegal