On misty New Year’s Eves of yore, revellers looked to a kilted Andy Stewart to lead them through the witching hour on television. Stewart’s songs about marching through the heather did not always hit the spot (“Has anyone ever tried marching through the heather?” remonstrated fellow Scot Billy Connolly), but the show captured a place in the nation’s heart.

Subsequent ringmasters of the ritual never quite succeeded and at first sight it seems strange that Stewart’s role has been taken up so assuredly by Jools Holland, a joanna-basher from south London who has just celebrated his 12th Hootenanny special for BBC2. On an evening when four-fifths of us were reportedly rejecting parties and staying at home, Holland is more of a fixture in our seasonal idiocy than ever.

Whereas Stewart and his White Heather Club tapped into the public’s affection for Highlands culture — or what passed for it — Holland is a pied piper who helps them to escape the insipid superficialities of modern pop and leads them back to authentic music that is both sophisticated and rollicking.

Living up to his reputation of being able to get anyone on his programme, on Friday night we saw the 46-year-old musician marshal a celebrity-encrusted line-up for an evening of infectious good cheer that featured Eric Clapton, Jamie Cullum, Natasha Bedingfield and the pipes and drums of an RAF band.

The annual bash displayed the big band leader’s contradictions, familiar to audiences of his weekly show Later . . . with Jools Holland, now in its 13th year. Guarded and confiding at the same time, he introduces performers of varying decrepitude and modishness with great deference, before sitting down at the piano to show them how it is done with a rattling display that would not disgrace Little Richard.

With a reedy south London voice that owes nothing to the Bells whisky he advertises (friends say it is not his tipple) and a waxy complexion, he seems an unlikely icon of cool. The working-class boy brought up in poverty is now married to the sculptor Christabel McEwan, ex-wife of Ned Lambton, the 7th Earl of Durham. He has two children, George, 20, and Rose, 19, from his first wife, Mary Leahy, and a 13-year-old daughter, Mabel, from his second marriage to McEwan.

Holland has come a long way since becoming a star on The Tube — a 1980s equivalent of Ready, Steady, Go — from which he was suspended for six weeks after blurting out the plug: “If you’re a groovy f*****, you’ll watch The Tube.”

He escaped most of the flak for the Millennium Dome’s opening night, of which he was musical director. But he may not have been forgiven by Jacques Chirac for a fiasco at the G8 economic summit in 1998. When Holland and his Rhythm & Blues Orchestra launched into their version of All You Need Is Love, Chirac understandably mistook the opening bars for the French national anthem and rose to his feet. Confused, Tony Blair, Bill Clinton and Boris Yeltsin stood up, too. Then they held hands and began dancing.

Those looking for a concealed personality behind Holland’s stage act are disappointed. His long-standing acquaintances say he has changed little since his days of playing in pubs. “He’s thoughtful and open-hearted,” said one. “Whenever I’ve been in darker periods of my life, he’s always been the first to pick up the phone and offer some assistance.”

He has an eccentric side and an addiction to classic cars. His collection features an Aston Martin DB5 Convertible, a 1958 Bentley S1, a Jaguar XK120 and a 1973 Ford Transit — a reminder of touring with his band in the 1970s, his first trip abroad. “There were four of us all squeezed in the back,” he recalled, “but there were no windows, just a tiny round hole on the side of the van and we’d take it in turns to look out and observe Europe.”

Holland’s influence is felt throughout the pop world. His show has showcased such new talents as Catatonia, Travis, Macy Gray, the Stereophonics and the Fugees. It was on Later . . . that Robbie Williams sang his hit song Angels and was astonished to look round and find Bonnie Raitt backing him up on slide guitar.

Nobody turns down invitations to record with him at the Greenwich studio in southeast London that he modelled on the purpose-built Welsh village of Portmeirion, the set of the cult 1960s television series The Prisoner. Here he has enlisted such diverse talents as Smokey Robinson, BB King and Robert Plant of Led Zeppelin for his Small World records. Recently Holland teamed up with Tom Jones in a successful collaboration.

He was born in Hammersmith, west London, on January 24, 1958, but was raised in Battersea, south London, in such impoverished conditions that the family once went without electricity for a year because they could not pay the bills.

In 1988 his father Derek, described as a marketing consultant, was jailed for 15 months for stealing antique jewellery worth £35,000 from Lady Durham, Jools’s future wife. He had found them in a recording studio beneath his flat and sold them to a London jewellers, spending the money on trips abroad.

Holland said later: “I think he went a bit soft in the head, but my motto is, ‘Forgive and forget’.”

Jools’s musical skill emerged early. He was taught to play the piano at the age of eight by his uncle Dave. Both his grandmothers were keen pianists, although he claimed he never heard them play because “after I was eight, and got my fingers on the keyboard, nobody got near it”.

To his youthful ear the big band sound held its own against contemporary pop. “I was listening to the Beatles and Motown, and I loved the sound. But at the same time, at 10 or 11, I was going back to the early R&B stuff, to Fats Domino or Little Richard, and thinking they were really great.”

Failing to shine at school, he was expelled at 15 after wrecking a teacher’s Triumph Herald. He maintained that in trying to retrieve a coin from beneath the vehicle, he released the handbrake, pushed hard, and the car vanished down a deep incline.

Although invited back to take his O-levels, he had already found his métier performing in local pubs for £10 a night, and got together with two friends, Glenn Tilbrook and Chris Difford, to form a band called Squeeze. “I first met him in 1974,” Difford said. “He turned up in a black leather jacket on the back of a motorbike, looking every bit the Marlon Brando type. He played some fantastic boogie-woogie piano, then fell off the piano stool and was sick on the floor.”

Working against the grain of 1970s punk, Squeeze’s cheery pub-band sound struggled for gigs, but they minted two hits, Up the Junction and the still popular Cool for Cats.

After five years Holland left to form a new band, Jools Holland and his Millionaires, but it was short-lived. Then his manager, Miles Copeland, suggested that he fronted a documentary about the group the Police. His confident voiceovers led to a job co-hosting The Tube with Paula Yates. They were the super-cool presenters of the 1980s and when she died of a heroin overdose in 2000 he was heartbroken. “However awful she might have been to other people,” he said, “she was always the most wonderful, loyal friend to me.”

He went on to host a revived Juke Box Jury and spent a brief period in America as co-host of an NBC show called Saturday Night. But it was Later . . . that cemented his television career and public profile, reinforced by tours with his 18- member big band.

“The thing about us is, we’re really a rock’n’roll group fused with a big band,” he explained once. “Maybe that’s why people like us: because the music has that drive to it that you don’t often get with a jazz orchestra.”

One day in 1994 the band were pulling into Arbroath when they heard on the radio that Andy Stewart, who came from the town, had died. “The band got very worked up and said to me, ‘It’s fate — you’re here to inherit his mantle’,” Holland recalled. “I don’t know, though. I couldn’t manage that cocked eyebrow of his.”