I ARRIVED on the political scene about 20 years ago armed with my toolbox of market research techniques and, with other modernisers, joined the battle to persuade a sceptical Labour Party of the value of listening to the public.

Since then, like all the main parties, my party has used polling research to provide an insight into the views of different voting groups, to help to develop and communicate policies, and to evaluate their success.

Labour was at first sceptical because it was worried that research might be abused to peddle a particular prejudice or point of view rather than providing insight. Any pollster who sells an idea instead of listening and evaluating will come out with faulty conclusions. Poor or skewed research results in lost elections, so methodological rigour is essential.

However, for a media eager to adopt modern polling techniques as well as find an exciting news story, there is no such inhibition. Over the years I’ve provided polling data to journalists, so I understand the tension between creating a good story and “telling it like it is”. I’m concerned that, increasingly, the polling commissioned by newspapers is being used to manufacture headlines rather than uncover facts. Look at two very different recent examples:

First, an ICM poll for The Guardian last month contrasted voters’ views of Gordon Brown and David Cameron, with the headline “Brown feels the Cameron effect”. ICM is a highly respected polling company and I have no quarrel with the methodology used. However, for a poll comparing two potential Prime Ministers to be of any value it is important to know that the attributes on which they were tested were the same qualities by which voters judge the job.

Sadly, this reality check did not happen. In the reporting of the poll most weight was given to “having a pleasant personality” — not on a prime ministerial job description I’ve ever come across — while “making the right decisions when the going gets tough” — a much more salient attribute — gained little prominence. On that quality, Mr Cameron trailed behind Mr Brown by 7 per cent. Other polls also show Mr Brown leading Mr Cameron in key areas — he has, for example, a 30 per cent advantage on who would best manage the economy. A poll by Opinion Leader Forum and Populus, published today, reveals that among swing voters Mr Brown has a 27 per cent lead over Mr Cameron as preferred Prime Minister.

Another example was the recent Newsnight “people meter” session — the talk of the Labour Party Conference even before the results were broadcast. Its top-line spin — that John Reid, not Mr Brown, could be a front runner in the race for the Labour leadership — was reported as truth, seized on gleefully by those whose cause it helped, and used to cast doubt on those potential candidates who fared less well. But its method — a hybrid of “ media pulse” analysis and a focus group — is controversial.

The focus group is a widely practised technique: a small group of people, usually six to eight voters, have a discussion moderated by a market research professional. Ideally, the voters are from a similar background so that the group dynamic allows for everyone to speak out truthfully. By contrast, Frank Luntz’s research for Newsnight featured 30 or so individuals drawn from a wide range of backgrounds. It was impossible to moderate fairly — we can see in the film that only the more assured or the loudest tended to contribute, with the effect of leading the whole session into “group think”. This flaw was compounded by the repeated use of shows of hands; secret ballots would have given the quiet dissenters a way to better express themselves.

Further bias was introduced in the dial test (the “media pulse” analysis). This is a tried-and-tested technique used in commercial market research, especially for ad testing. Each participant is given a dial to turn up or down indicating simple approval or disapproval in response to material shown. It is crucial to be clear about what message is being tested.

But the Newsnight report confused measuring reactions to the content of speeches and reactions to the character of each politician shown. While Mr Reid was seen making a speech with a populist anti-terror message, Mr Brown and David Miliband were both answering sensitive questions about the Labour Party leadership. Are the voters responding to the man, or the message? A previous Luntz comparison for Newsnight gives a clue. He showed a focus group a Brown Budget speech (heavily scripted, heavy-going and delivered in an antagonistic House of Commons) and Mr Cameron’s unscripted, light-hearted and rapturously received speech to last year’s Tory party conference. Given this unfair comparison, little wonder that the focus group gave the thumbs up to Mr Cameron.

Yet this work was reported as if it provided real insight, even though it contradicted much of the published poll data (ICM the same week gave Brown at 46 per cent, a 28 per cent lead over his nearest competitor — Mr Reid) in answer to the question: “Who do you think should be the next leader of the Labour Party and hence Prime Minister?” Mr Brown’s lead is consistent and unchanged month by month. One thing I’ve learnt over the years is how long it takes to change people’s views for good or bad; that conclusion, however, makes for unexciting copy. It also takes a long time to establish a profile. In focus groups conducted by Opinion Leader Research, voters shown a photograph of Mr Reid identified him as Ming Campbell.

Accurate and insightful market research depends not just on reliable fieldwork but also on thoughtful and experienced analysis that sets the findings in a proper context and interprets them accordingly. Too often the analysis is carried out with an eye on the headline, not an eye on the truth.

Deborah Mattinson is joint chief executive of Opinion Leader Research

Times Recommends

• In the real world, the public sector must pay
• Playing politics in a land of double Dutch
• Be like the plinth people: not set in stone