On stage in a crowded meeting room in Central London in 2006, I impulsively decided to go out on a limb in front of a crowd of 200 online marketing professionals and make another celebrity ballroom prediction. In the States I had developed a reputation for analysing reality television contests by measuring contestant popularity by the volume of searches for those celebrities on engines such as Google, Yahoo Search and MSN Search. It was eight weeks before the finale of Strictly Come Dancing but the search volume for Mark Ramprakash was off the charts. A lot can happen in eight weeks, but the overwhelming data meant that I just had to make this one prediction.

Here at Hitwise, with an aggregate and anonymous sample of more than 25 million internet users worldwide, and more than eight million specific to the UK, we have the ability to analyse several million search terms each week. As search engines become more integral to our lives, the aggregate volume of searches on specific terms provides insights into how events in the outside world affect us. Reality television predictions were a proof of concept.

The idea of predicting reality television show results using search data first occurred to me earlier that year while watching the American show Dancing with the Stars. As the host, Tom Bergeron, described the voting process (a portion of the vote was determined by audience calls, texts and web votes) I reasoned that searches for celebrities participating in the show should correlate with their popularity and, since the contests were very well produced popularity contests, I should be able to predict a winner.

My first prediction, during the 2006 season of the show, was based on the impressive volume of searches for Stacy Keibler, a leggy blonde female wrestler. Unfortunately, that prediction didn't go quite as well: Keibler took third place. In a post mortem we determined that searches for Keibler centred more on quests for “hot pictures” than her dancing prowess. By factoring in search intent (what we internally refer to as a Stacy Keibler correction coefficient), our predictions have become increasingly accurate. This online/offline correlation was the first step in finding what other insights were possible.

Starting with simple observations, we've studied searches for diets, which reach their yearly peak on the first week of the new year, before crashing only a week later. Conclusion: our commitments to shed pounds are remarkably short-lived. We can also gain insight into perceptions of world events.

What's commonly referred to as cognitive dissonance describes our inner conflict of answering questions honestly versus painting ourselves in the best possible light. When giving lectures I have an extreme method of illustrating how cognitive dissonance can lead to dramatically different results.

When speaking to large audiences across the US, I ask, by show of hands, who regularly visits adult entertainment websites. To date, after asking that question to several thousand conference attendees, not one person has raised their hand. In other words, based on this sample, the internet is porn-free. When we analysed the aggregate behaviour of ten million anonymous internet users, more than 11 per cent of all website visits are going to sites in the “adult” category. Somebody's not telling the truth. While this may be an extreme example, cognitive dissonance can distort our understanding of much more mundane activity.

How would you answer the question: how has the economy changed your online behaviour? Would you answer that you've become more obsessed with celebrities? In America we noticed that, as the economy worsened in the autumn, the most affluent internet users decreased their visits to online brokerage sites, preferring instead to frequent celebrity blogs such as Perezhilton.com and TMZ.com. Here in the UK, celebrity obsession has taken a more financial bent with the “money-saving expert” Martin Lewis garnering more searches than the usual Britney Spears or X-Factor contestants.

In the US, apart from the expected increase in visits to employment, financial news and discount coupon sites, internet users are increasingly engaged in diversionary activities, visiting online game, social networking and movie rental sites, using the internet as an escape from troubling financial news.

Apparently the diversions of the British internet user during these trying times are more liquid in nature. According to Hitwise, searches for afternoon tea have increased by 67 per cent over the past 12 months. In the past four weeks there were more than 1,500 variations of “afternoon tea” queries. Surprisingly, the venues garnering the most searches were high-end establishments such as Claridge's, The Ritz, and Fortnum & Mason. The most visited restaurant site is Beer in the Evening (beerintheevening.com).

As our economic challenges continue into unprecedented territory, aggregate search data is likely to provide us with a new perspective on changing consumer sentiment - as well as reality television winners. If you don't follow Strictly Come Dancing, the 2006 winner was Mark Ramprakash.

Bill Tancer is the author of Click - What We Do Online and Why it Matters

Times Recommends

• We have kit and leadership. Let’s go
• Behind the scenes at the biggest deal
• Conferences — what a waste of money