Governments don’t tend to win by-elections. Especially mid-term; especially mid-term in a third term. And the main opposition party doesn’t tend to get relegated to third place, either. Thursday’s results were encouraging for Labour as the party’s performance beat even the heady early days of new Labour. This was the lowest swing against the government since 1997.

The results also compare favourably with John Major’s first posttransition by-election challenge in Ribble Valley, where the Conservatives lost the seat and their vote dropped by 22%, despite relatively positive national opinion polls. Ealing Southall dented Labour’s vote by just 7%, even though the turnout fell by more than 13%.

The results in Ealing Southall and Sedgefield are deeply disappointing for the Tories - relegated to third place in both. David Cameron threw everything at the Ealing campaign: he visited the area an extraordinary five times and even put his own name on the ballot paper: this wasn’t just the Conservative party, it was David Cameron’s Conservative party. Tony Lit was Cameron’s hand-picked candidate, too, chosen to reflect his leader’s own personal style. And the voters rejected him.

What went wrong? The Tories’ Ealing campaign lost out because it misjudged the mood. It offered showbiz and razzmatazz at a time when Britain is feeling sober and reflective. It was style without substance and the style set the wrong tone.

People believe Britain faces new challenges. They are anxious about crime, the fight against terrorism; they want further improvements in the National Health Service and schools; they want their children to get a foot on the housing ladder.

Against this backdrop we have seen a dramatic and positive shift in Labour’s fortunes. The party enjoys a clear opinion poll lead on all substantive policy areas and, interestingly, we find that this advantage grows significantly across the board when the party leaders’ names are mentioned. It is this that underpins Labour’s lead in voting intentions.

People ask how this turnaround has come about, given that Cameron tended to be ahead of Gordon Brown on the so-called hypothetical question used by many pollsters before the transition to anticipate reaction to Brown’s leadership. I think there are two explanations: one is that interviewees find it almost impossible to guess how they will feel at a future point when circumstances have changed. The “hypothetical question” was never likely to be a reliable predictor of people’s behaviour.

More importantly, though, recent polls reveal which leadership characteristics really matter. People tend not to be fickle and their views change slowly and incrementally. Their assessment of the main party leaders is consistent - they have known all three for some time now. What has changed is the relative saliency of the characteristics that they judge leaders by. It is about being the right leader for the time.

When Cameron took over as leader it was important for him to be manifestly different: his fresh and youthful image pressed the right buttons and contrasted positively with an old-fashioned Tory party and a Labour government that appeared to be running out of steam.

Cameron has not changed but the context has. The new government that he now faces led by Brown seems purposeful and focused. It looks different. The apparent drift that voters were reacting against has been replaced by energy and action. Brown’s personal attributes, established through a decade at the Treasury, define the job that he now does. These leadership qualities - strength, experience and conviction - match the popular mood and chime with people’s sense of what matters for a prime minister. So while Cameron looked fresh when up against Tony Blair, against Brown he is found wanting.

Looking at polling on leader personality traits published over the past year or so, Cameron has consistently trailed Brown on characteristics such as “sticks to what he believes in under pressure”. Cameron’s shortfall is now translating into a growing concern that he might not be the best person for the job. Recent polls show Brown’s lead to be eight points over Cameron on “would make the best prime miister”; last week’s ICM poll showed Labour under Brown with a 26-point lead over the Tories under Cameron on “best equipped to govern Britain”.

A year ago the Tories were able to achieve a landslide victory locally in Ealing when they won Ealing borough council. Last week, on the same turf and despite a hard-fought campaign, Cameron’s Conservatives failed in their bid to topple Labour – or even to dislodge the Liberal Democrats from second place.

The local nature of by-elections makes them electoral novelties and voting patterns can sometimes conflict with wider national trends as voters make their point in a risk-free environment. On this occasion, however, we saw voters in Ealing Southall sending an important message to the Conservatives about their leader’s approach: slick communication skills may have been enough to put him on the map a year ago, but the electorate has moved on and such slickness is simply not enough now. It is a message the Tories will ignore at their peril.

Deborah Mattinson has been joint chief executive of Opinion Leader Research for 15 years. She is currently setting up a company that will do political research and campaigning. She is an adviser to Gordon Brown.

Times Recommends

• Chelsea Flower Show has lost the plot
• The age of personal vitriolic abuse
• Obama: the new Great Redeemer