It’s one of those gatherings typical of local elections. A poster has gone up, advising black voters to register to stop the British National party getting any seats, and various candidates for mayor of London are uniting beneath it for a photo shoot.

At last Boris Johnson, the Tory candidate, shambles up, looking tired and dishevelled. His shave isn’t much better than the Green party girl’s make-up (which looks as though it was put on by a clown in the dark).

As they wait around, Boris and Ken Livingstone – the current mayor - exchange a bit of friendly banter, like a pair of Second Division football managers before a local derby. Then everyone lines up under the new poster, and various worthies in dog collars make hyperventilating speeches about fascism and what an awful advertisement for London it would be.

The next poster along, I can’t help noting, is for London Pride beer – “hops, barley, yeast and pride” – which seems to sum up the candidates in the mayoral race. Boris is hops, the Lib Dem Brian Paddick is barley – barley here, barley registering. The Green girl is yeast – fizzy, and a little goes a long way – and Ken is pride: lots of pride and perhaps pride before a fall.

A television reporter asks Ken whether he minds sharing the antifascist platform with Boris, who once wrote of “piccaninnies and watermelon smiles”. Ken says graciously: “This is a time for candidates to stand together.” Boris looks niggardly.

Everyone here is going through the motions. “Politicians agree on voter registration” isn’t going to make the front page or the six o’clock news. Each candidate will leave as soon as it’s polite to do so.

I notice that Boris doesn’t move anywhere without his minders: a pair of fly, fashionable young men with the whiff of last night’s nightclub about them, both wearing flash overcoats. They are a type I usually associate with New Labour. Since Boris's campaign was taken in hand by the Australian spin doctor Lynton Crosby, the word is that he’s being closely managed, gagged and strait-jacketed – and these are coves who do it.

Maybe they’re getting to him. Today, I notice, there is something strangely awkward about him: not the awkwardness of a dyspraxic teddy bear walking upright – he’s always been like that – but an uneasiness that comes from within.

We all depart, leaving Paddick to talk to the last film crew. Their tripod isn’t tall enough to get his head in, so he’s doing the half splits, like a giraffe waiting to be mounted. The man is a proper plod pratt.

Later I try to catch up with Boris at Middlesex University, where he’s going to be talking at the business school. We might as well be on the outskirts of Paisley: it’s a huge muddle of hastily expanded blocks, as impenetrable as the Orinoco. After my fifth tributary corridor, I finally find Boris being shown round by various suits with name tags. The place is crammed with students and looks like an open casting for a Benetton ad.

In the little lecture room Boris is introduced and we are told that he will talk about diversity and then take questions. He stands up nervously, holding a page of scribbled notes. Really he ought to have memorised a set seven-minute speech: he’s going to have to give it enough times. He stumbles on: diversity is apparently a good thing, and so on and on, but something is very wrong.

I’ve heard him speak many times; I’ve spoken to him often; I’ve shadowed him during elections; and I’ll admit to liking him, admiring his journalism and the general thrust of his humane liberal jib.

The thing about Boris is that he’s good at performing the one-man show that is Boris. He has himself down pat as a comedy turn: the timing, the self-deprecation, the touch of quixotic bombast, the heart in the right place, the hair all over the place. This, he can do in his sleep – it’s what’s made him the most popular Tory MP among people who don’t have much time for Tories or MPs.

However, here in this small white room, with students who ought to be eating out of his hand, he’s doing a bad impression of himself. His voice is rocking up and down regardless of content. He is swallowing the ends of sentences, stopping, losing his place, shouting when he loses the thread – and none of it is amusing or even particularly interesting.

At one point he says we shouldn’t all be like Peer Gynt and his onion. You what? Then he tells us about witnessing one of new Labour’s cringe-making citizenship ceremonies and finding it very moving. The old Boris, the real Boris, would have blown away a citizenship ceremony, with its flag and picture of the Queen and cup of tea, as wholly risible and pathetic. He maunders on, listing the things that people have told him are typically British: liberty, um, freedom, um, fair play, um, um, restraint, um, irony, um, and a whimsical sense of humour, um, um and a certain talent for wordplay.

The audience look at him with blank incomprehension. They’ve lived here all their lives, and the thought of standing up and cheering for whimsy and palindromes is even more foreign to them than the countries their grandads came from.

And then Boris introduces us to his grandfather. His Turkish Muslim grandad, it seems, is going to be dug up for this election and exhibited more often than Tutankhamun. “I wonder what he’d say,” Boris asks rhetorically, “if he could see his grandson standing for mayor of this great city . . . ” And the students, overwhelmingly second-and third-generation immigrants, know exactly what their grandparents would say: “Get a proper job: study accountancy.”

The questions afterwards are smart and thoughtful. Boris agrees with everyone: cheaper houses, no student debt, nicer immigration officials at Heathrow, more money for everything.

A Dutch girl asks: how come tolerance is such a big political idea when, on a one-to-one basis, saying “I tolerate you” is not unequivocally happy and huggy. It’s a good question, and it should be right up Boris’s street, touching as it does not only on individual freedom but also on wordplay – but he ducks it and rambles.

When it’s all over, after the ritual tilt at bendy buses, the 200 students give him a polite round of applause – but it’s been an underwhelming and nervous performance.

As Boris is walking out, he spots me and smiles. “I was just talking about you. You remember when you followed me round last time? You said I was the worst politician in the world . . .”

You are, I say. Then: I think you lost them with Peer Gynt and the onion. “No, no, they all understood Peer Gynt and the onion. Didn’t you?” he shouts across the room. There is a murmur of nos and lots of shaking heads. “Ah,” he says, and punches me.

As we walk out together through the numberless corridors, he’s back to his old avuncular, clever and observant self. “This is the best job in politics, isn’t it?” he says. “This is the really big one.”

And you might get it, I say. “Yes, yes.” He has a nervous, concentrated look, as you might have while waiting for the all-clear siren to sound. “I might – it’s very close, isn’t it?” He flicks me a sideways glance. Yes, I think it is quite close. What are the three things you’ll do when you take over City Hall on the first day: what’s at the top of your agenda?

“Um, um.” He’s plainly never considered this; like the Americans in Iraq, there is no plan for after victory. He searches the horizon for inspiration: “Um, well, put conductors and policemen on buses. Yes, and take away free passes from children.” That’s my son you’re taking about, I point out.

“Only if they’re naughty – your son isn’t naughty, is he? Um, er, give pensioners 24-hour travel passes. Er, stop Tube workers going on strike, bring in legislation to, oh, er.”

Allowing octogenarians travel passes to go clubbing doesn’t quite have the ring of a mission statement. Tell you what, Boris: have a think and call me tomorrow.

That night, while Boris is having a ghastly, unfocused, underprepared, stuttering and blustering factless rant at Paddick and the mercurial Livingstone on Newsnight, I briefly meet the Tory leader, David Cameron, at the British Press Awards.

“Boris is doing very well – he might even win,” Cameron says. He sounds both rhetorical and inquisitorial, but also as if he might be surprised if I agree.

The truth is nobody really thought that Boris could win; the Conservatives just didn’t want a repeat of the humiliation they went through with the last Tory candidate, Steven Norris (and don’t even mention Jeffrey Archer). If Boris could acquit himself in a gentlemanly fashion, knock up a reasonable score, draw some blood, that would be enough: there would be tea and biscuits and a junior cabinet post back at Westminster.

The only person who thought Boris had a chance was Livingstone. And now, as the polls show, there’s not much more than a margin of error between them, and everything looks different.

The received wisdom is that Ken will lose the election for himself, and that the best Boris can do in the meantime is to keep his feet out of his mouth. In short, the less he can behave like the Boris everyone loves, the better – which seems to be an odd strategy. And Boris doesn’t look or sound happy with it.

The next evening, there is a big hustings at Methodist Central Hall, Westminster – “Don’t call it a hustings,” says the press officer. It’s apparently being held by a coalition of churches, mosques, temples, schools, colleges, youth clubs and trade unions.

“It’s not a hustings,” the press officer insists again. “In fact, I’d much rather you write about our agenda than what the candidates have to say.”

Really? I’ll bear that in mind. Tonight Boris gets roundly booed by the trade union contingent and there is palpable multicultural bias towards Livingstone. Ken plays to the gallery for all it’s worth, which may be quite a lot, and promises to get the Venezuelan president, Hugo Chavez, to pay for Oyster travel cards for long-term failed asylum seekers. At this, I was the only person in the room who laughed out loud. Boris just looked glum.

Before he went, he gave me a moment. “You wanted my first three things I’d do in office. Um, er, accountability – I’d put all expenditure on the net. Er, two, police on the buses. Three: scrap The Londoner newspaper and plant trees all over London. Er, stop paying anarchist and communist collectives and all that rot, and then wi-fi – wi-fi everywhere, I’ll wi-fi, wi-fi everything.” He looks up at the great Wesleyan dome and the huge pipe organ of the Methodist hall and grins. It’s a moment of old Boris revelation.

So what’s gone wrong? He’s used to rubbing along with an insouciant air of amateur hopelessness, of turning up in old plimsolls and serving underarm with a girl’s racket – so when he scores, it looks effortless; and when he doesn’t, people love him for seeming not to care.

However, this is a really huge prize, unquestionably the best job in British politics, with more power and money than most cabinet ministers. And Boris has suddenly woken up to the fact that it could all be his. He can almost touch it. Like Antony on the barge, he’s racked by the thought of what he must do to get it; and the thought of losing is just too painful to contemplate. To win, he has to commit everything to the fight. He’s caught on the horns of his own ambition and his terror of failing.

Peer Gynt was an interesting illustration to pull out of the air; for, as Freud said, there are no accidents. For those of you who have forgotten the central theme of Ibsen’s play, it’s about how a man must strive to be truly and honestly himself. When Gynt peels the onion to try to find its central core, that which makes it an onion, he discovers that the layers don’t hide the truth. They are the truth.

Boris has two weeks to stop peeling his onion, dump the dour Scandinavian doubt and get on with being himself.

Times Recommends

• Tree stump 'shows image of Virgin Mary'
• BBC drops Strictly judge for younger model
• Trafalgar Square 'reduced to a spectacle'