The one place Gordon Brown is likely to feel at home when he takes his holiday in the Conservative heartland of Southwold is waist-deep in the sea. In a town where a beach hut can fetch £80,000 and a fish supper costs more than a tenner, the icy brown breakers and sole-puncturing shingle would seem reassuringly austere. And Mr Brown is likely to be in a particularly austere mood, after the by-election drubbing Labour received in Glasgow last night.

Even on a cloudless July day this week only single-minded holiday-makers emerged from behind their windbreaks to venture into the bracing chill of the North Sea. If Mr Brown were to risk such a public spectacle it would be the most courageous moment for a Labour leader on a beach since Neil Kinnock toppled, fully clothed, into the surf during a photo opportunity at his party’s conference in Brighton in 1983.

Residents of Southwold and nearby Walberswick, the two closest settlements to Mr Brown’s holiday destination near the Suffolk coast, were in one voice when asked about the Prime Minister’s visit: he will not find himself among friends, but he should be left alone to enjoy himself.

The most hostile reception may be at Southwold’s Swan Hotel, where some guests who had booked rooms telephoned with the intention of cancelling until they were reassured that rumours of Mr Brown staying there were false. No guest has complained about a forthcoming visit by the Prince of Wales next week.

The Prime Minister is likely to receive a warmer welcome in Walberswick, where celebrity residents come in a variety of colours on the political spectrum. Sir Clement Freud, the broadcaster and former Liberal MP, and Martin Bell, the white-suited former independent MP, both have homes near the village. Other homeowners include Paul Greengrass, who directed The Bourne Ultimatum, and Richard Curtis, who is married to Sir Clement’s daughter, Emma.

The village, where residents offer home-grown courgettes for sale at the roadside with an honesty box for payment, is a chocolate box come to life. Wild roses curl around doorways of fishermen’s cottages and children cast lines for crabs from a footbridge on the way to the beach. If Mr Brown takes the two-minute drive through the village he will pass two pubs (The Bell and The Anchor) a pair of tea-rooms (The Parish Lantern and The Potter’s Wheel) and a handful of boutiques with names like The Old Curiosity Shop.

Even the wildlife is prosperous in Walberswick. The bird table at The Bell spills over with unpecked brown bread. Paul Ireland, the landlord, would welcome the Prime Minister, he said. "As long as he bought me a pint. The amount of taxes I pay here..." He trails off, suggesting that a pint of Adnams, the local ale, would settle any ill feeling.

Mr Brown and his family will not be starved for entertainment, judging by the village notice board. A poster for Family Bingo promises cash prizes, "continuous bingo from 6pm to 8pm" and "home-made snacks". He may wish to avoid the Conservative Fete on August 9, but he would be welcome this weekend at the Walberswick Horticultural Show, at which there is a women-only shortlist for the Iris Blois Memorial Cup. Entrants will vie to grow the best rhubarb and beetroot, and bake the finest savoury tart.

The fiercest competition may be in this year’s special category: ginger nuts.

Southwold’s Electric Picture Palace, a cinema built in 2000 but with affectations of a bygone era, is catering for an audience of a certain age with Swallows and Amazons and Shall We Dance, a film starring Fred Astaire and Ginger Rogers described as featuring "some amazing roller-skating".

The Summer Theatre, also in Southwold, is preparing for the end of its run of Salad Days, the twee 1950s musical in which graduates sing "piano" so that it rhymes with "guano", and will raise the curtain on Murder Mistaken, a melodrama on the topic of inheritance.

Sue Doy, an usherette at the theatre said that Mr Brown’s political abilities should be forgotten during his visit. "I think he should be left alone on holiday," she said. "He’s no difference to any of us."

The only visible preparation made for the Prime Minister is an enclosure drawn in the sand on Southwold beach marked "reserved for Gordon Brown". Pip Piper, a 48-year-old nurse from Norwich, claimed responsibility but said she was not an admirer of Mr Brown. "I was going to kick sand in his face. I’m very upset with Labour. I expected more after ten years. He has got to come out and be generous with the people of Southwold. If he did come down here and had a swim I’m sure he would enjoy it. If it was him and Sarah and their two children then they would have a great time."

Times Recommends

• How I lost the gift of joy for a while
• Why Ann Widdecombe has got it wrong
• Alas, it’s the end of the road for petrolheads