‘I know who you are,” said Ken Livingstone, sounding like a secondary modern Mr Chips recollecting a former pupil. “Would you like to know what my favourite restaurant is?”

“Very much,” I say; Ken used to be a restaurant critic himself.

“It’s Vasco & Piero’s on Poland Street. Lovely, I’ve been going there for 30 years.”

He closes his eyes in reverie. “They send a van to Italy every week and come back with fresh ingredients; mozzarella, truffles, wonderful stuff. Would you like their phone number?” And he reaches into his breast pocket and pulls out an address book, like a teenage girl’s – small and falling to bits, rammed with neatly written phone numbers. He has obviously had it for years.

This is a man who, for eight years, has been in charge of one of the most powerful cities in the world. London has a GDP bigger than Switzerland or Sweden. Its mayor has more hands-on day-to-day power than any other politician in the country. And he has this tatty, pre-Filofax, predigital, 01-for-London address book. It’s a tellingly touching – and weirdly out of touch – way of remaining in touch.

I have been trying to get in touch with Ken’s campaign all week. We’ve had three journalists, who really should have had better things to do, begging to find out from the mayor’s election staff where he was going to be and what he was going to be doing there, so that I could tag along.

I asked these three to keep a log of their abortive attempts to contact the mayor’s campaign team. There were dozens of unreturned calls and unanswered e-mails. Aren’t they fighting an election for public office? Surely voters are entitled to know what Ken is up to? What on earth are they so afraid of?

Press avoidance seems to be part of Ken’s paranoia. He seems unwilling to engage with the media unless it is through his own party newspaper or sycophantic hacks. But without the oxygen of media coverage, this campaign, which started out being gripping, has slumped into a glutinous stasis that is barely reported at all.

A high turnout is traditionally good for Labour. Ken used to be the great communicator, the man with catchy soundbites and outrageous headline-grabbing assertions: Mr Dogma. But now he is strangely equivocal and piano. He shrugs more often than he points. He’s torn between his distrust and dislike of the press and his need to galvanise voters.

At 9am on Wednesday we finally get a tip-off (not from his office) that Ken will be in Peckham in half an hour talking about, well, what do politicians go to Peckham to talk about – green spaces and municipal flower planting? Traffic calming and traffic light synchronicity? Of course not. They come here to talk murder. What pies are to Melton Mowbray and Shakespeare is to Stratford, so premature death is to Peckham.

Even the taxi driver, who dropped me off without actually stopping, said: “That’s the stretch you want down there, mate, where they killed that Damilola.”

This is a borough where the local landmarks are police tape and piles of flowers and teddy bears. Ken is coming to some sort of centre on the high street. He’s late.

I notice that Ken Livingstone is always late. Lateness is part of his charm. Punctuality may be the politeness of kings, but Ken’s unapologetic tardiness shows that he’s a man of the people. It might also be because he insists on taking public transport.

The drop-in centre is, as far as I can work out, part Citizens Advice, part community centre, part police depot, part youth club and young offenders’ chat room. According to the notices in the window, it has been supported by at least a dozen quangos, several charities and a few NGOs. When some poor kid gets stabbed, this is the sort of community project where they dump the conscience cash.

In charge is a charismatic black lady with a lot of gold teeth and attitude. Ken’s minders, some selected press and a few kids traipse into a tiny recording studio in the basement. Recording studios are apparently what youth clubs have now instead of ping-pong.

The gold tooth lady tells us about gun crime. Ken nods and replies with a list of all the money and resources and initiatives that he has put into knives and guns, delinquency, hooliganism, drugs, boredom and everything short of bringing back conscription.

A couple of deeply embarrassed kids are encouraged to get into the booth and rap with him. They shuffle and mumble and look mortified before someone chants the first line of White Christmas – which is unintentionally funny.

Then a beefy lad with bad skin declaims his story.

“I was a footballer. Then it went wrong. I was sitting in a pub and a bloke came to hit me and then I pushed him. He hit his head and died, unfortunately, and I was in prison for 10 months. Now I’m here turning my life around. I want to thank them for standing by me when I got so angry in Belmarsh and my picture was all over the papers.”

The lady with the gold teeth gives him a hug. Ken shakes his hand and congratulates him on turning his life around and the rest of us shuffle uncomfortably. It is altogether a weirdly Dickensian tableau. The sinner lost and found, the moved politicians, the general air of righteousness and great philanthropic work.

Ken leaves with his minders and walks to the train station. I still don’t really know what it is we’ve just been doing here and apparently neither does he. He looks oddly mal-coordinated. He has pigeon toes and big feet and walks with a cartoonish dopiness. His arms don’t synchronise and he’s got that eternally cheap cotton suit and shapeless flasher’s mac with the final style addition of a girl’s rucksack.

Altogether with his little head bent forward he looks like an oversized special needs kid.

I hang around in Peckham for a bit. Like most Londoners I’ve never been here before.

If I want to go and look at murder sites I’ll do the Jack the Ripper heritage trail in Aldgate. Actually Peckham turns out to be rather wonderful: bright and lively, proper cafes and pie and mash shops, a great long market street full of butchers with boiling chickens in the windows and grocers with piles of African tubers and strange wilting green things. There are also wonderful garish clothes shops blaring music, all colour and life. And not a single chain store. I recommend it for weekend breaks.

But me and Ken aren’t finished. I’ve got a date with him that night in Soho. I’m told to wait outside a pub on Old Compton Street. It’s freezing. A fistful of lesbians are handing out Ken for Mayor beer mats. (How is it that virtually every fashion designer and style arbiter in the world is a gay man, but that lesbians collectively have come up with Morrissey as a style icon?)

Ken’s late. Again. He maunders up Old Compton Street to a smatter of applause and some flashers and then leads us off on a stroll around gay Soho.

His press officer Victoria (the one who wouldn’t return any e-mails or calls) and his agent (ditto) are now both immensely attentive and charming, telling me how pleased they are I could come and how they’d love it if I could join them at a private party in Heaven. (The office has been trying to get access to this do on my behalf for a week with no joy; it kept being told that it was a “strictly private party”.)

We pause reverently outside the Admiral Duncan pub (it was nail-bombed in 1999 by a neo-Nazi, three people died and 70 were wounded) and then we troop off.

The reaction to Ken in the street is muted. After-office drinkers look up without interest, occasionally someone yells. It’s difficult to tell if they’re being encouraging or ironic. There are plenty of shouts of “wanker”.

Ken waves and every so often a drunk lad will come up and ask him to fix their lives, the buses or the weather. He promises to do what he can. Facts and figures tumble out of him with alacrity – if not conviction. He’s like an automated phone system. Press one for bendy buses and Boris; press two for immigration and the Olympics; press three for gay and lesbian issues; stay on the line for a smooth operator.

Except he isn’t that smooth any more. There’s something about Ken’s whole demeanour that looks beaten. He’s losing from the inside out. He was always a grey man, the Stalinist version of John Major with a whiny south London accent. But now there’s a weariness, an absence of enthusiasm.

The exchanges are ritualised. He goes into his performance of slagging off the Tory candidate Boris Johnson, adding just a whisper of a full stop each time before mentioning the BNP. All the bravado’s gone, though. The gesture politics, the solidarity with the laughably irrelevant Third World causes. It’s like he’s reading off a menu of stuff he doesn’t want to eat any more.

It’s really close, says his agent; really, really close. We stop off at the Gaydar multi-storey club where we are joined by Chris Bryant, the gay MP for Rhondda, who coincidentally was outed for being photographed in his underpants on Gaydar’s pickup network. Upstairs the topless Brazilian waiters hand round drinks that are made of champagne, cream, brandy and something else. Ken falls on his with gratitude. A man comes up and kisses me. It’s my hairdresser, Vladimir.

Then we’re back in the street and down Frith Street. More bars. Gin and tonic for Ken. His welcome is underwhelming: people shake his hand and ask a question, but don’t seem to care about the answer. It’s hardly Dick Whittington.

And we’re back in the street and circling round again. “Where are we going?” asks Ken. “We’re going in circles.” And when he thinks no one’s listening, he mutters: “It’s a bit like this campaign.”

The caravan’s got smaller, the camera crews have had enough, as have most of the photographers. There’s me, a couple from a Greek paper, the editor of the Pink Paper and a few others.

We find ourselves back in Old Compton Street. An hour’s drinking has made all the difference. The response to Ken is now raucous and not kind.

We traipse on through the dark to a pub off the Strand that I’m told is popular with gay civil servants. A very proper man chatting up a less proper lad asks me pointedly if I know how long Ken’s planning on staying.

Then the mayor is off to Heaven, the ancient and venerable and passé gay nightclub. It’s nine o’clock and they’ve opened the VIP section specially for Ken to make a speech. We don’t refer to gays and lesbians any more – it’s LGBT: lesbian, gay, bi and transgender. It’s an obviously awkward acronym for Ken: he keeps making it sound like bacon, lettuce and tomato.

His speech is fluent enough; he looks wistful and talks about the good old days when homophobia was there to be slain along with racism. The days when he could talk about women’s rights and troops out of Ireland and solidarity with the printers – when he had a moustache like the Village People. He’s like a skinny Falstaff in his cups, remembering yesterday’s battles – the great heroic fights all behind him now.

The young Hals in the audience have gone on to work in banks and advertising, the law and the civil service. They don’t know this old confrontationalist braggart any more. The gay man next to me whispers that he’s fed up with being patronised and patted on the head.

“We’re not a put-upon minority,” he says, fiercely voting out gender politics. “We care about all the same things all other Londoners care about: buses and drains and policemen.”

Ken’s coming to policemen. “If some of you want to vote for Brian Paddick because he’s the first openly gay mayoral candidate, I quite understand. But give me your second vote.” This sounds both patronising and defeatist.

He goes on a lot about the second votes as if he already imagines the first-choice ballot to be lost.

The old gay campaigner Peter Tatchell rings me to say that Ken has been a very fair-weather friend to gay Londoners. He hasn’t funded things he said he would – particularly overtly gay projects and the gay football World Cup which only needs £10,000. Perhaps he’s spending a lot more time courting Muslim communities because he thinks they’ll vote en masse. He never seems to confront the homophobia of the more fundamental mullahs with whom he insists on appearing on platforms.

For all Ken’s bright new dawns and visions of a homogeneous, vibrant, multicultural People’s London, his politics is still about splitting communities up into their simple ethnic, sexual, social or religious components. Blacks and gays, immigrants, Muslims, single mums, pensioners, disabled, Tories. It’s the Balkanisation of London into manageable groups who can be made to feel both vulnerable or special. Who can be frightened or bought off. Each in turn having to come to the great father at City Hall for largesse and protection.

But London’s not like that any more. People don’t think of themselves in those kinds of boxes. Things ain’t what they used to be. Racism ain’t what it used to be. Class ain’t what it used to be. Gender ain’t what it used to be.

Ken’s old socialist, crusading, divisive, fist-waving politics is over and passé. He doesn’t have any new verses for the old song. In eight years he has overseen immense changes in the city, almost beyond recognition; and he’s been responsible for good and bad. But like Routemaster buses and telephone boxes and pearly queens, he’s been left behind by the pace of change. He doesn’t have any new answers. He doesn’t even have any new questions. He’s just running on nostalgia.

Times Recommends

• Deal struck to rescue Jodrell Bank
• S&M madam: Nazi claims are rubbish
• Operation Goodnight: holiday town curfew