Herman radiated mischief. He had a twinkle in his eye and a spring in his step, and his moustache was turned up at the ends, so his face had a permanent grin. He advised me to buy two sticks of dynamite, each sprouting a long, cartoon-style fuse. Now I needed a small plastic bag of pink granules: ammonium nitrate. Mix it with a drop of diesel, he explained, and you get a bigger bang.

He chuckled, as if this were a joke, but it wasn’t. We were stocking up on gifts prior to plunging into a Bolivian silver mine. It’s the custom to offer a little something to the miners. The explosives would be put to good use.

A mine tour is one of the must-dos for the more adventurous visitor to Potosi. Looming up behind the city, which claims to be the highest in the world, is the Cerro Rico – literally the Rich Mountain – a pinkish peak that has been producing precious metals since the Spanish conquistadors founded Potosi in the 1540s. Then, the mountain was 5,240 metres. Today it’s 4,824 metres and riddled with shafts and tunnels, like a Swiss cheese.

Silver from the Cerro Rico fuelled the Spanish empire. In the 16th century, Potosi was one of the world’s largest cities, rivalled only by London, Paris and Seville. I’d wandered its cobbled colonial streets, put my nose inside the baroque cathedral, and seen the fortress-like Casa Nacional de Moneda – the mint where the silver was turned into coins and ingots. The woman who showed me round grimaced when she told me Bolivian coins are now made in Spain.

In the old days, they carried the silver from Potosi over the Andes and down to the Pacific by mule train. I’d been up higher, acclimatising to the altitude at another of Bolivia’s superlatives – the biggest, highest salt flats in the world, at Uyuni. It is one of the most remarkable landscapes I’ve seen: infinite and perfectly even, the dry lakebed rock hard and blindingly white. Set against the crispest blue sky, it was just a dual expanse of brilliant colour.

En route down to Potosi, I’d passed other mining towns, snuffed out and left to crumble. Like Pulacayo, where I stopped to visit a cemetery for steam trains. Among them was the first locomotive to enter Bolivia, brought to replace those mules, and another that had been held up by Butch Cassidy and the Sundance Kid.

I remember learning about Bolivian metal mines in geography lessons at school; that’s partly why I was keen to visit one of Potosi’s shafts. The other reason feels more spiritual. Miners appear on both sides of my family tree. A few generations back, the Middletons were digging lead for a living beneath the hills of Derbyshire, and my wife’s grandmother ran a tin mine in East Africa – the first woman in Uganda to hold an explosives licence.

So, dynamite in hand, Herman and I moved up the steep road towards Potosi’s magic mountain. Our mine, known as Maria, is a cooperative, worked by 400 miners for silver and zinc. Several sat outside chewing coca leaves when we arrived. They were working up a wodge that is stored in the cheek, Herman said. It staves off fatigue and hunger, and also acts like a clock. The effect lasts for eight to 10 hours. When its juices run dry, you know your shift is over.

The entrance to the mine is just a small hole in the hillside. We had to bend our heads to get inside. I’d opted for the short mine tour, just an hour or two, and Herman had said I could leave when I got tired. But tiredness wasn’t a problem, because claustrophobia got me first.

A few steps in, while I was still adjusting to the darkness around the dribble of light from my helmet, Herman grabbed my forearm. He played his beam downwards on my boot – it was six inches from the edge of a black void. The chasm swallowed the light from both our lamps. “A shaft down to another level,” Herman said simply. I nodded, but my mind was still processing the lack of a handrail or barrier. No steps either – nothing to stop me tumbling into the vacuum. Except Herman.

Several turns further into the darkness, there came the dull thud of an explosion somewhere inside the mountain. A shimmer of reverberation rippled through the tunnel and I felt a trickle of earth fall on my hair.

It would have been nice to stand up straight, to take a deep breath and flex my muscles. Just to show myself I wasn’t afraid. But I couldn’t stand tall because the roof was too low, and a deep breath would mean a lungful of dust. Besides, I was afraid. Herman’s light disappeared round another bend. I hadn’t realised I suffered from claustrophobia, but I sure did now. I wanted out, but I wasn’t confident about finding my own way back.

Walking as fast as I could while keeping an eye out for deadly shafts, I caught up with Herman.

“How much further?” I asked. The highlight of most mine tours is a visit to El Tio, part devil, part miners’ protector. Every mine has one. Herman said we were close.

I pondered the idea of panic, but thought better of it. Panicking in such a confined space would probably compound the problem.

Turning another bend, we came upon El Tio. He was a plaster bust of a man sitting in his own side-tunnel. He had a cigarette in his mouth and the unmistakable bulge in one cheek. Like Herman, his face bore a permanent smile.

Coca leaves lay all around him in the dust. Miners regularly bring El Tio gifts. Herman suggested we offer him a toast, and produced a bottle of alcohol. I noticed it was 96% proof. My slug burnt all the way down, but supplied nothing in the way of courage. I suggested we might leave.

As we emerged into the blinding sunshine, Herman stretched his arms to the sky and told me that conditions for miners haven’t changed much since colonial times. I nodded, but was only half listening. I was concentrating on the fresh mountain air and the beautiful sky. My ancestors would not have been proud of me.

Nick Middleton travelled as a guest of Cox & Kings

Travel Details: Cox & Kings (020 7873 5000, www.coxandkings.co.uk) can tailor-make itineraries throughout Bolivia. A 12-night trip, visiting Potosi and the mines, also La Paz, Lake Titicaca and the Uyuni Salt Flats, starts at £2,550pp. The price includes flights from Heathrow with British Airways and American Airlines via Miami, B&B accommodation, private transfers and all excursions. Or try Trips Worldwide (0117 311 4400, www.tripsworldwide.co.uk ), or Sunvil Latin America (020 8568 4499, www.sunvil.co.uk ).

Times recommends

• Confessions at 39,000 feet - your stories
• Europe's 20 best Christmas markets 2009
• Cruise passengers say: ‘We’ll tip when we want to’