For a while in the 19th century, visits to Ashby - as most locals refer to this small town (population 13,000) in northwest Leicestershire - were all the rage.

In the 1820s the publication of Sir Walter Scott's Ivanhoe led to a mini stampede of tourists to the ruins of Ashby Castle, surprising locals who had previously regarded the remains as a rather handy source of masonry. Scott used the environs of the castle as the setting for the jousting tournament between the Black Knight and Ivanhoe, and the castle itself as a scene for “high revelry” with Prince John.

Visitors, enthralled by the story, came in their thousands and further damage to the castle, originally wrecked because it was an important Royalist stronghold during the Civil War, was blocked.

Now it is the main attraction in Ashby, with fine views from its half-destroyed tower. Up there, to the southwest, you can see the site of the former Ivanhoe Baths (lots of things are called “Ivanhoe” in Ashby), which was created in 1822 to cash in on growing local interest. For a while this flourished, but it fell into decline in the 20th century and the shell of the elegant building in which it was once housed was pulled down by the council in 1962.

“That was a great shame. John Betjeman pleaded its case,” says Robert Jones, one of the volunteers at the excellent little Ashby de la Zouch Museum, next to the tourist information centre. The museum displays explain the town's name. “De la Zouch” comes from Norman times, when a family by that name ruled the roost.

“Ashby” dates from earlier, when there was believed to be a settlement of Danes. Apparently there was once a Dane named Aski who lived here, and “by” was a Scandinavian term for settlement.

“The letters ‘by' in Derby are there for the same reason,” says Jones, who shows me a side-room with a poster from the 1952 film of Ivanhoe starring Elizabeth Taylor and Joan Fontaine, and information on Thomas Cook's package holidays from Leicester to Ashby to see the Ivanhoe Baths in the 1850s.

There is also an exhibit on the town's historic pubs. Many of these still prosper and, after buying a guide to Ashby's old inns, I go on a pub crawl, stopping first at the White Hart on Market Street, a run of fine old buildings including a quaint Victorian market.

Through a glass panel in the floor of the White Hart you can see a hole where an 18th-century landlord once kept a bear that he would set on rowdy customers. Back in those days it was known as “the bawdiest public house in the Midlands”, but it's not so rowdy on my visit and serves a good pint (as does the nearby Shoulder of Mutton and the Plough). There was no Ivanhoe pub, I was surprised to find. But that didn't matter. For a quirky weekend, Ashby was full of other surprises.

Need to know

Bed down at ...

The Royal Hotel (01530 412833, www.royal hotelashby.com), built in 1826, is next to the site of the old Ivanhoe Baths, with a grand façade with Greek columns and comfortable rooms. Doubles from £75, B&B. The Clockmaker's House (01530 417974, www.clockmakershouse.com) is a terrific little B&B with double rooms from £65; the best is the Blue Room at the top.

Chow down at ...

The Mews Restaurant (01530 414444, www.mewsrestaurant.co.uk) on Mill Lane serves good modern British cuisine in a candlelit room tucked away off Market Street. Two courses £14.95. If you only have time for a snack, try a sandwich at the News Café in the Victorian market.

Outside bets ...

Calke Abbey (01332 863822, www. nationaltrust.org.uk) is a wonderful 18th-century country house in 600 acres, about four miles north of Ashby. The house is preserved exactly as the National Trust found it when taking over in 1986; tickets £8. Or try Snibston Discovery Museum (01530 278444, www.snibston.com), four miles to the east, with interactive displays explaining technology to children and above-ground tours of a former coal colliery; entrance £6.75.

Times recommends

• Dubai cheaper than Doncaster as room rates tumble
• World Cup 2010 - how to follow England
• Copenhagen: Europe's greenest city