I saw it coming, but the blow to my face still smarted. I struggled to avoid gulping as my nose streamed, and for a moment I panicked. In front was the San Francisco skyline. To my right the Golden Gate Bridge poked out through banks of fog, while behind me stood Alcatraz, the notorious prison from which there was no escape.

The hit wasn’t from some angry hoodie – it was from a wave. I was in San Francisco Bay, whose treacherous currents, chilly waters and marauding sharks meant that in Alcatraz’s 150-year history as a prison only one escaping inmate had made it to dry land.

As unusual tours of California go, swimming the one-and-a-half miles from “the Rock” to San Francisco’s shoreline easily tops the list. Local “aquapreneur” Gary Emich, who has done the trip 510 times without the aid of a ferry, now organises the crossing for enthusiastic swimmers who want to see “the City by the Bay” from a unique perspective. I was attempting it with two California-based friends, Dan Shevchik and Rick Dewey.

At least that was the plan, but the notorious local fog wasn’t playing ball. “If we don’t see the shore, we don’t go, and we’ve got a five-minute window with the tides,” was the support boat captain’s cheery assessment as he looked at his watch (7.55am). We bobbed around just off the rocky shoreline of the former maximum-security penitentiary. Dan and Rick’s parents and Rick’s wife, on another boat alongside, flaunted glasses of Buck’s Fizz in our direction.

Miraculously, as I was about to launch into my “I haven’t come 7,000 bloody miles just to sit in a dinghy,” speech, a skyscraper poked out of the mist, followed by another and another as the sun showed itself at last. “Go!” said Gary with an air of urgency. We went.

Alcatraz was a military prison from 1859 to 1933. Then, until 1963, it was a federal jail, housing inmates such as Al “Scarface” Capone, Alvin “Creepy” Karpis, George “Machine Gun” Kelly and the Birdman of Alcatraz, Robert Stroud (who never actually kept birds there – that was at Leavenworth Penitentiary). “You are entitled to food, clothing, shelter and medical attention” said the rule-book given to inmates on arrival. “Anything else you get is a privilege.” Over any 24-hour period there were at least 13 headcounts. Cells were cold, small (5ft wide, 9ft long, 7ft high) and offered no privacy. No wonder some tried to escape.

In 1962 Frank Morris and brothers John and Clarence Anglin put dummy heads in their beds, crawled along a ventilation pipe at night and then, using improvised floats made from raincoats, jumped in the bay. They were never seen again, presumed drowned. Later that year John Paul Scott made it to land, using inflated gloves for extra buoyancy, but was picked up when he reached the Golden Gate Bridge, too exhausted even to pull himself on to rocks.

And now my turn. Billy “the Fish” Hide, facing the currents (crossings are timed to avoid the worst), the cold (I had a wetsuit) and the sharks. Ah, yes, the sharks. Every – and I mean every – local I spoke to about my swim mentioned them, as did the guide on Alcatraz during a tour two days earlier. Only a few weeks before a great white had chomped a surfer near Monterey, and another had decided to use a sea-kayak as a toothpick – both men survived. Truth be told, none of the species found in the bay is a man-eater, the most common being the sand shark.

I plunged in and immediately panicked, starting to hyperventilate. It wasn’t the temperature (the water was bracing, but I’ve endured far worse in Scottish lochs), but the lack of visibility and the gloomy, murky pea-green colour. Who knew what lurked beneath?

I struck out. A large swell meant that breathing on the left was fine, but getting air on the right had to be timed carefully or else I’d end up with a mouthful of seawater. It was some time before I was able to calm down and realise that this was actually quite enjoyable. Dan and Rick, both former members of the Harvard University swimming team, were evidently enjoying themselves, speeding up then returning to granny Hide at the back, stopping to take pictures with a waterproof camera. Gary, also swimming with the group, guided us in front, his yellow cap acting as a beacon as we neared the shore.

“I always tell people to stop for a minute halfway through their swim,” he said as we surveyed the scene. “Just let it really sink in, where you are and what exactly you’re doing. You’ve got infamous Alcatraz with its ruins and prison and cliffs. To the east are the Oakland hills, to the west the bright orange Golden Gate Bridge and Marin Headlands, and in front the beautiful San Francisco skyline. So few people ever get this perspective. It’s a mental video that will stay with you the rest of your life.”

I had to agree. A sort of elation was sweeping over me, now that I had calmed down. Open-water swimming is such a liberating experience, an escape from the constraints and monotony of the pool. I resumed my stroke, head down, looking into the void, the only distraction the bubbles of air produced as my hands cut into the water. This was, like, totally awesome and gnarly dude, as we don’t often say in North London.

In just under 55 minutes my feet touched the sandy bottom and I walked on to the beach near the corner of Jefferson and Hyde streets, slapping backs with the others and looking forward to getting my hands on a growler, which, Rick assured me, was a jug of beer.

If you’re a halfway decent swimmer, and are equipped with a wetsuit, the indispensable knowledge of someone like Gary and a backup boat, there is no reason why you shouldn’t be able to plan your own escape from Alcatraz. And the best thing, aside from the unique views of the great city and the sense of achievement? Well, the bragging rights down the pub are absolutely enormous.

Need to know

Will Hide travelled to San Francisco with British Airways Holidays (0870 2433406, www.ba.com), which offers five-night stays and flights from £620pp. Book before September 25.
Gary Emich (001 650 359 3773, www.lanelinestoshorelines.com) organises Alctatraz swims with 65 days’ notice; £175 for up to five people. Swims operate year round, but the water is warmest in late August to mid-October.

Further information: www.nps.gov/alcatraz and www.escapefromalcatraztriathlon.com.

Or try these...

The World Winter Swimming Championships are being held outside Finland for the first time next February – in South London (www.slsc.org.uk).
Swim around Rottnest Island in Western Australia (www.rottnestchannelswim.com.au).
Cross the Hellespont from Asia to Europe (www.swimtrek.com).
Crawl around Manhattan . A full circuit is 46km (www.swimnyc.com).
Try the iconic swim, across the English Channel (www.channelswimming.net).

Browse travel offers at Times Selects

Times recommends

• The world's 10 weirdest things to eat
• Correspondents: hoteliers revenge for Tripadvisor
• A slow, ponderous journey up the Mekong