The pilot kills the engine and soon we are rocking in the waves. Alcatraz is 15 yards away, my starting point. A mile and a half across the bay lies the prize of San Francisco — a thicket of skyscrapers shimmering in the early morning haze. It’s the moment I’ve been dreading, but one hundreds of inmates once longed for — a chance to swim to freedom.

The powerful mythology of Alcatraz refuses to die. When fog is not shrouding the bay, the prison’s implacable architecture of cell house, water tower and lighthouse are clearly visible, an evocative reminder of the most notorious prison island of all.

Named by a Spanish explorer after alcatraces — gannets — it spent most of its inhabited years as an American military fort, and prison. Only in 1934 did it become the infamous federal penitentiary, and then for just 29 years before high running costs led attorney general Bobby Ken­nedy to order its closure.

By then, its reputation was assured. Forty-five years on, it attracts more than 1m tourists a year — and the previous day, I had been one of them. As with a great cathedral, walking through the cell house transports you to a different age, but for the real Alcatraz experience, you have to make your own escape.

I strip off and feel the tart Pacific breeze on skin protected only by a pair of Speedos. This has been the hardest decision of all — to forgo the reassurance of a wetsuit. Already I have the advantage of an accompanying boat, its pilot and Gary, a local swimming guide who has planned my route against the strong tides.

Wetsuits insulate you from your surroundings, but if I’m going to escape from Alcatraz, I want to know what it feels like. At least, that was the plan, but now, looking at the 16C, slate-grey water of San Francisco Bay, I can almost feel the chill leeching through my bones.

We wait as a tanker lumbers past, then Gary tells me it’s time. Stinging and now fully awake, I swim to the “beach”, which is covered by the high tide, take a few steps on the submerged pebbles and touch the cliff where I’m to start from.

There were 14 attempts, but only one man, John Paul Scott, is known to have definitely reached land. And when he clambered onto the rocks beneath the Golden Gate Bridge, he was so weakened — it was December, and he had been swept away by strong tides — that he was taken to hospital before being returned to Alcatraz.

On the tour, former inmate Darwin Coon had aimed a good put-down at my attempt: “It wasn’t the swim that kept us from doing it, it was those guns and watchtowers!” Yet the bay is crucial to Alcatraz’s escape-proof reputation. At only a mile and a half from land, distance was not the problem, and neither were the sharks the authorities liked to talk up.

While there are great whites in the nearby ocean, San Francisco Bay is home only to harmless varieties. No, the real battle is against the tides and cold. And the key to the Alcatraz myth is its location: isolated in the middle of the windswept bay and at the same time within sight and sound of one of the world’s most free-spirited cities.

I trot back into the bay and start to crawl away from the island, seeking rhythm in my strokes. The water is a greenish hue and I can see no more than a foot or two ahead. After a fitful night dreaming that fog would separate me from the boat, cramp would disable me and a shark would finish me off, it’s a relief to be swimming.

I am by no means a pioneer. Gary estimates that about 5,000 people a year do the swim, most of them locals.

Times recommends

• Oasis of the Seas: the Times review
• The 20 best new year party places to stay
• Green Spaces 2009: are you the winning writer?