What would the late Queen Elizabeth Queen Mother have made of America electing its first black President? Sadly we will never know but, after reading Ed Stourton's musings this week, I don't think we can assume that she'd have cracked open a celebratory bottle of gin. Stourton, a Radio 4 Today presenter, spent a few hours in her company in the 1990s and left of the opinion that she was “a ghastly old bigot”.

After lunch at Windsor, he remarked that he had just returned from an EU summit and, fixing him with a twinkly smile, the Nation's Favourite Granny replied: “It'll never work, you know. The EEC. It will never work with all those Huns, wops and dagos.” Had she been a barmaid bellowing such sentiments across a crowded pub, our national treasure would have been taken down the police station faster than you can say “Johnny Foreigner”. But because she was elderly with “old-fashioned views” (she apparently insisted on calling Zimbabwe “Rhodesia” and wasn't keen on black majority rule), we evidently think that it's no big deal.

Some commentators even suggested that Stourton was the villain for being ungallant enough to mention something said in private to sell his new book. As Hugo Vickers, the royal commentator and author, said: “The Queen Mother lived in another age and she did say these sorts of things. Trying to put her into a box of political correctness is as silly as trying to put the sea into the sky.” Ah, that's all right then.

I am confused by our readiness to excuse older people who come over all Alf Garnett because they are “from a different generation”. I thought human beings were supposed to wise up with age. The young are from a different generation, too, but we forgive them nothing. They only have to appear in public wearing a hooded fashion top and we call the police.

Let's be clear: it's understandable that people who have lived through the two world wars and lost loved ones can be left with a lasting mistrust of former enemy nations (the Queen Mother disliked Germans, having lost her brother on the Western Front). And we should never forget that people like this were fighting to prevent the most grotesque racism and that we owe our forefathers a huge debt of gratitude.But that doesn't mean that some elderly people don't sometimes say things that make you want to eat your hands with mortification.

I'll never forget my friend's cuddly granny once remarking that she loved Coronation Street. And do you enjoy EastEnders too, I asked politely? “Oh no, too many black faces,” she said, sweetly, “I see enough bloody darkies hanging round the precinct as it is.” Stunned into open-gobbed silence, her grandson and I let it pass. A septuagenarian, who had seemed a lovely neighbour when I was living in London, once stopped me in a state of fury to confide that “four coons” were moving in across the road and was there anything that I, as a journalist, could do about it? Another friend's much-loved grandmother refused to speak civilly to a Kenyan nurse who was caring for her - and would check her purse as soon as she had gone.

I'm not suggesting that all old people are prejudiced and that younger people aren't. This week a BBC presenter of a mere 40 was sacked for telling a taxi firm not to send an Asian driver to collect her daughter. But the elderly tend to be less inhibited about saying what they think.

Age Concern in North Wales was so shocked three years ago by the racist attitudes of some of those at meetings that it asked them to tone it down. “We are frequently dismayed (sometimes disgusted) at the discriminatory attitudes of some older people, in particular concerning ethnicity or racial origin,” it said in its newsletter.

“We feel it cannot be right for an organisation such as ours to fight discrimination, while at the same time doing little or nothing to combat prejudice and discriminatory attitudes among older people.” One couple had said that they moved to Wales from London simply to get away from black people - though those weren't the actual words they used.

So what's the right response when an older person has an Alf Garnett moment? Obviously younger people have them as well, but that's easy - you can call them four-letter names or simply get out of their taxi. (Oh, just kidding. I'm not suggesting all taxi drivers are racist.) With old people you don't want to be disrespectful so you change the subject. We're taught from childhood to treat the elderly with courtesy but when they say things that could land a younger person in court is it OK to turn a deaf ear?

Andeliza Tucker, from Liverpool, was recently charged with racially aggravated harassment for allegedly calling her Irish neighbour a “f***ing leprechaun”. You can make up your own mind about whether the CPS was right to try to prosecute (the case was eventually dropped) but I'd bet my house that if Tucker had been 80 and not 18, no one would have taken it seriously. Imagine the outcry if Prince William had, like his grandfather, asked Aborigines in Australia: “Do you still throw spears at each other?” As it was, it just became the stuff of semi-amused headlines.Maybe some people of a certain age don't see “light-hearted” racism as being as harmful as other generations, raised on political correctness, do.

Perhaps Bernie Ecclestone, 78, genuinely saw nothing wrong with the Spanish spectators mocking Lewis Hamilton with blackened faces and curly wigs. “I think it's all nonsense. I don't think it was anything to do with racism...I think people look and read into things that are not there,” he said, before realising that the rest of the world didn't quite agree.

Yes, so easily misconstrued, that blacked-up face and curly wig ensemble. Still, Ed Stourton, if you're passing Windsor Castle in the next few weeks, I'd seriously consider wearing one. Purely for disguise purposes.

Times Recommends

• How to stop the Queen picking the next PM
• To save the planet, strike a deal with Big Oil
• Bankers think it’s all over. Time to think again