On many occasions, the organisers of the Hay-on-Wye literary festival, which is held in a field near Wales every year, have invited me to go along and give a talk. And on an equal number of occasions, I’ve said no.

There’s a good reason for this. You might imagine that Hay is a lovely day out for all the family, a chance for children to meet the authors they love and, conversely, an opportunity for writers to meet the people who actually read their books.

Of course, it’s no such thing. Mainly it’s a chance for ramblers and hippies to gather in a field and convince themselves that everyone thinks the same way that they do. In essence, it’s a competition each year to see who can dream up the most organically idiotic way of cleaning their teeth. Cow manure or nettles. The great debate.

Then there are the attractions. There’s lots of movement and dance, a carbon gym and plenty of unnecessarily funky capital letters from the 2FaCeD DaNcE workshop. This year The Guardian had even built a House of Hay (geddit) out of what in Farringdon Road would undoubtedly pass for hay, but was, in fact, straw. It’d be an ideal building material for people who like rats and want to have breathing difficulties.

I have always thought that if I went along they’d pull my hair and steal my milk in the playground. But this year, with Boris in London’s hot seat, a Tory looking after Crewe and Gordon scoring nought in the opinion polls, I figured – wrongly, as it turned out – I’d be safe. So I fired up the SUV and said: “Come on, kids. Let’s go and laugh at the lefties. It’ll be fun.”

Annoyingly, the organisers had not sent any directions for those who were coming by road. Instead, we were told to use something called a train that would take us to Hereford – 20 miles shy of the books.

If you didn’t fancy this, you could come by coach. There were two options. A six-day trek from London via every market town in the land. Or an even longer journey from Bradford to Worcester, which is listed in my road map as being “nowhere near Hay-on-Wye”. What you were supposed to do if you were coming from Crewe, it didn’t say. Stay at home and watch Tom Clancy films on your plasma, probably.

To get around the festival itself, shuttle buses were provided. And, of course, these were very publicly running on biodiesel. Or, as I like to call it, a poor man’s lunch.

If you didn’t fancy assuaging your middle-class guilt with that, you could use a bicycle, and three people had done just that. But because it was raining heavily, most had simply come in their stupid little eco-Fiat cars and turned 84% of the surrounding countryside into a quagmire. Green? No, more a soupy Ypres brown, in fact.

Inside the tented village, many of the organisers were wearing tie-dye. One chap was sporting a kaftan. Beards were everywhere, and everyone was squelching around in sturdy shoes from the Street-Porter range. It looked like a scene from the Haight-Ashbury happy-clappy handbook on bonkers living.

But no. Behind the scenes, there was trouble afoot. Because the festival is now sponsored in part by a bank, and because all bankers, obviously, are the spawn of Satan, there’s now a rival festival a mile or so down the road.

This event, organised by someone calling herself the Poet, had invited Arthur Scargill to speak while the assembled druids ate bits of dirt from their smocks and mocked people at the real Hay festival.

So far, then, no books had actually made an appearance, but this came as no surprise since it is written that when one or two socialists come together they will immediately forget about the common goal and start squabbling over who’s got the most environmentally friendly yurt. It’s the Judaean People’s Front all over again.

To try to give my children a taste of what life might have been like if their mum and dad had been lunatics, we bought them an ice cream made from sheep’s milk. Nutritionally, they’d have been better off licking Arthur Scargill’s hair. We did at least run into the children’s author Georgia Byng, who took my youngest daughter’s mind off the shit sheep ice cream.

My talk seemed to go quite well. The tent was full of families who’d paid £15 a ticket, none of which comes to me, incidentally; and so, in return, I tried to give them all a laugh – which they were unlikely to get from the ice cream or the Mexican diplomat’s lecture on his conservation project to save the Latin American monarch butterfly.

But with each answer, I was inadvertently signing my own death warrant. There I was, jokily telling all the small boys in the audience that I’d once done 186mph through the Limehouse Link in London, and that speed limits are for the weak. And backstage it was all being moulded by the eco-greens into a howling, sack-the-idiot press release.

I probably will be, and it’s my own silly fault. I never saw it coming. I was expecting them to burn a pile of my books; I was ready for George Monbiot to leap on stage and arrest me for having a patio heater. I’d even taken a change of clothing in case a fat woman, full of root vegetables and hate, shoved a custard pie in my face again. It never occurred to me that I’d been invited specifically to shoot myself in the face.

Top Gear is back on your screens in three weeks. It’ll be hosted by Bill Oddie and will feature lots of movement and dance. And how you can make a car out of straw.

Times Recommends

• Preening populism has put us off democracy
• In 2009 as in 1931, debts must be repaid
• Here’s where all the women get off