The exotic scent of patchouli hangs over Cahors’s most secret garden. Two years ago, the Jardin Mauresque was an unloved scrag of land sandwiched between a school and a tenement building, at the end of a skinny lane.

Today the green square behind the slatted gate is planted with banana palm and lemon balm. There are aromatic herbs in lines of sea-green pots, and somewhere in the shade of a languorous albizia silk tree, a small fountain plays a watery tune.

The Moroccan Garden is now a long way from unloved — it is one of Les Jardins Secrets, a new collection of 35 postage-stamp plots pasted into assorted corners of this consummately medieval French town, which receive visitors daily from all over Europe.

It is somewhere to ponder the intoxicating aroma of the Love Generation, and cool down away from the unremitting heat of a Quercy summer.

Do the Lot in France: 10 winter treats

• Le travel blatting: the French art of the car

Beziers: gateway to the real France

Not prettified, not twee: for a dramatic drive into the authentic Midi, make for Beziers

• Hamming it up in Bayonne

There's more to Provence than sun-bathing

... as Stanley Stewart discovered when he took a horde of kids to France who demanded art, history and activities

• The best towns in Provence

Beach and lobsters in Ile de Re

Don't be put off by the rain on the Ile de Ré - the balmy Atlantic island hides many treasures

• Life is a carnival in France

The Jardin Mauresque is number 23 on my official plan of Cahors’s secret gardens, and I aim to track down a couple more in the grounds of the Cathedral of St Etienne before lunch, when the sun will summit the sky above the Lot Valley and chase everyone into a shady cafe or bar.

First, the Monks’ Vegetable Garden, featuring foliage grown for the pot in the Middle Ages — things such as flowering purple artichokes, borage, red-veined sorrel and marigold. Then on to the Herbarium, the medicinal garden, where the plants are grouped together according to the ailments they treat.

I first came to Cahors in a canary-yellow mini 30 years ago, probably with my own supply of patchouli oil tucked away in my suitcase. I was enchanted by the town’s shady lanes of redbrick buildings, the plane trees that shade Boulevard Gambetta, its main artery, the pedestrian lanes in the medieval quarter, and the slowness of the Lot.

Hemmed by limestone cliffs and meandered around by the river, Cahors started life as a fortified Roman town and grew rich and comely in the 13th-century. It still has 300 very special buildings of that vintage, and the triple-towered Pont Valentre is considered the best medieval bridge left in Europe.

But mostly I remember being seduced by the town’s inky “black wine”, and the truffles that feature on most menus, grown amid great secrecy in nearby Lalbenque.I wasn’t the only one to succumb: in the 19th-century Cahors outsold Bordeaux, and was the preferred choice of the Russian tsars, who liked their wine to be as omnipotent as their rule.

This time I’ve arrived by train via Paris, and I’m staying at the Hotel Terminus across the road from the station, one of those elegant 19th-century properties with floor-to-ceiling windows and art-nouveau detail.

The Terminus is ideally situated to view the secret gardens — starting just a short walk away, with the dainty display of “petite” plots along the banks of the Lot. Some of these have been donated by other cities, others created by noted horticulturists. I follow the leaf-shaped steel badges set into the pavement to the first garden on my list, the Jardin d’Ivresse — Garden of Inebriation. No wonder it’s number one here — it’s a vineyard.

Big or small, secreted behind walls or just fringing the street on a strip little bigger than a window box, all the Secret Gardens are themed. Some, it must be said, are a bit contrived — such as those depicting the Seven Deadly Sins. “Lazy” is an empty flower bed and “Lust” an amorous couple in plaster of Paris getting it on in the potting shed.

Times recommends

• 'New York has lost its edge'
• The UK and Ireland's 25 best Christmas events
• Chalet Chef 2010: the winners