When is a joke not a joke? The answer is simple - when no one laughs. Comedy is quite different from other art-forms - if people don't enjoy a sad film, they might call it sentimental, mawkish or boring. But they don't suggest that it isn't sad. We just think it's not to our taste. We have a completely different reaction to a failed joke. No one hears a line that fails to raise a laugh and says, yes, that's me, I have no sense of humour. We don't think it's subjective. We take refuge in the same attack: it's not funny. If the joke doesn't work for us, we deny its very existence as a joke.

Russell Brand and Jonathan Ross are experiencing the aftermath of a joke gone horribly wrong. What do you get if you cross a Spanish waiter with ill-judged prank - 10,000 complaints and rising. After a series of on-air phonecalls to Andrew Sachs, the 78-year-old actor, about a liaison with his granddaughter, the two face calls for their resignation, an Ofcom inquiry, even a suggestion that they be prosecuted. Why are the British public, normally so proud of their sense of humour, not laughing?

The first issue is power. The dynamics between comedian and audience are complex. An audience are happy to be complicit in all kinds of unpleasant jokes, as long as they don't feel bad. Essentially, the rule is that laughing at the rich and famous is comedy, laughing at the poor and disenfranchised is bullying. In the past few years there have been a plethora of jokes about Heather Mills on primetime television. Because she's a wealthy celebrity, audiences are far more comfortable laughing at her disability than they would be about another woman injured in a traffic accident.

A similar shift has occurred with racist jokes - most people now balk at jokes predicated on someone's ethnicity, but nationality is still fair game. The comedy circuit is full of people who wouldn't dream of joking about Asian or black immigrants, but don't hesitate to joke about Polish plumbers. When Anne Robinson consigned the Welsh to Room 101, or The Simpsons derided the French as cheese-eating surrender monkeys, we were quite happy to see it broadcast on TV. Robinson faced a few calls for an apology, but it didn't exactly harm her career.

Interestingly, it seems to be Sachs's age that has caused most disquiet. A straw poll of my colleagues suggests that even other comedians don't think it's funny to badger an elderly man. Yet we've all heard jokes about John McCain's age without being worried. Again, it's about power. Mr McCain has thrust himself into the public eye. If he wants to have his finger on the nuclear button, goes the reasoning, he should be able to take a joke. But Sachs is someone people remember fondly - and he wasn't looking for publicity. I wonder if the number of calls has upset people. One stupid prank call might have been forgivable; four messages of increasingly insincere apologies less so.

The second issue that has caused so much concern is that the show was prerecorded. As someone who once used the F-word on Radio 5 Live (after midnight, but still), I know that BBC audiences are willing to forgive a slip of the tongue on live radio, particularly if you apologise instantly and unreservedly. But the fact that a producer heard this show and thought it was fine to broadcast seems to have baffled commentators.

This is a problem with television and radio that live comedy doesn't face. If you make an off-colour joke on stage, the audience tends to respond instantly - they gasp, or boo, or even walk out. I once told a joke about Siamese twin babies at a gig in Chertsey that caused a third of the (admittedly tiny) audience to leave. One of them then hammered on the window wailing until her husband dragged her away. It didn't stop me from telling the joke again, but it certainly convinced me that I should judge my audience carefully before I tried it another time.

An audience act as a valuable barometer of what is and isn't OK to make jokes about. It's fine to pursue a tasteless joke if it bombs once or twice, but if it only ever bombs, it's probably just not funny enough to counter its offence. I once crushed a heckler who was trashing my gig with his random shouting, and only later realised that he was mentally ill and a war veteran.

The audience knew immediately that I was wrong - some were delighted that I could be such a bitch, but most were appalled. I still feel bad about it years later. It's difficult to replicate that audience barometer with a small production team - what seems funny to four or five people who know each other can miss the mark completely for millions of listeners.

Brand is a terrific stand-up, and a large part of his act is toying with his audience's expectations of what is or isn't appropriate. It's hard to see what he loses from a scandal: he has a young audience who won't easily be deterred. The censure of Ross is far more virulent - is that because he has daughters?

I find it rather depressing that in all the furore, no one other than Sachs himself seems to have thought to offer or demand an apology for his granddaughter.

Natalie Haynes is a stand-up comic and writer

Times Recommends

• Caesar & Cicero: Balls & Mandelson
• The anti-sex brigade are the worst
• Small act of charity won’t change the world