You might reasonably think that when you search for a subject on the web using, say, Google, your results would be a round-up of the most relevant and informative sites. But you’d be wrong. A shady group of computer experts known collectively as “black hats” is trying to ambush your results in order to show you links to websites that have little to do with the search you had in mind.

Black hats, so-called because of the hats traditionally worn by Hollywood baddies, are responsible for an escalating cyber-war in which huge sums of money are at stake. If the black hats can divert you from the web page you would expect to land on to another in which they have a commercial interest, they take a fee. Multiply it by the millions of searches carried out every day and the rewards become clear.

The problem is so serious that companies such as Google are being repeatedly forced to change their complicated search-ranking system to try to outwit the black hats. Every few weeks it does the “Google dance”, which means it changes the rules by which search items are listed.

Douglas Merrill, a vice-president of Google, likened the problem to an arms race. “They (black hats) are highly motivated. There is a lot of money at stake,” he said.

In the early days black hats were low-rent hackers, working from home and pushing Googlers towards sites advertising the three Ps — porn, poker and pills. But as web searching has grown so have the commercial opportunities and big companies are using their services to clamber up the rankings.

Black hats study the way search engines work and look for ways to skew the results to promote certain websites in the search ranking. For example, black hats will try to distort searches for “horses” so that instead of being directed to equestrian centres people find gambling sites. They do this by a variety of means (see panel) but one way is to generate thousands, sometimes millions, of spurious web pages and link them to a master site. This fools the search engine into thinking that surfers are being directed to a particular site that is an authority on a subject and that site is given a high ranking.

Of the 30m new websites set up each month, 28m are bogus sites created by black hats designed to fool search engines into sending visitors to pages with adverts from which they benefit, according to MessageLabs, a cyber security firm. “The practice is growing every day,” said Mark Sunner, chief technology officer.

The true value of a high search ranking was highlighted this year when leaked AOL figures suggested only one in 10 web users delved past the first 10 results on Google.

So how do black hatters operate? Aaron Wall, 27, runs a website called www.blackhatseo.com. This murky practice is not illegal, he insisted. “Who is and who isn’t a black hat is dependent on what Google says is black hat,” said Wall. “They would certainly class me as a black hat.”

He claimed that some of his peers made well into seven figures a month. “Wages can change quickly,” said Wall. “My first year online I made nothing, but since then I have made a few hundred thousand dollars.”

Wall said he learnt the tricks of the trade by comparing notes with other geeks. “I started off doing lots of spammy stuff (web spam),” he said. He moved on to buying old and trusted website addresses purely for placing advertising links. He likened subverting search results to a high stakes poker game. “It’s like playing a hand of cards against a competitor with their cards showing . . . You just got to figure out why and how to win.”

Wall insisted that almost everything about black hatting was common sense. “The search engines always have some algorithmic holes which just about anyone can exploit. But you have to have the right mindset to be able to see algorithmic flaws, and then you have to have a bit of business sense or creativity to leverage them into profit.”

As the stakes get higher, search engine companies are fighting back by demoting in their web rankings — or sometimes removing — anyone they suspect of black hat tactics. BMW was recently accused of using underhand techniques and its German website was briefly delisted from Google.

Google said the car maker had used a false page stuffed with key words to pull undeservedly large online audiences. The claim was strongly denied by BMW. “The delisting didn’t affect us much because very few of our visitors come from search engines, but what hurt was the damage to our reputation,” said a BMW spokesman.

“They used our name to raise awareness, to set an example. By the time it became public we had already removed the stuff from the website.”

Tricks of the black hat trade

Key-word stuffing Plastering a website with words likely to be perceived as relevant by a search engine’s web-crawling tool, known as a spider

Cloaking A website showing different pages to a spider from those displayed to surfers. Cloaked pages on BMW’s German site got it into trouble with Google

Buying ‘real estate’ Snapping up older, trusted websites and peppering them with links to a client’s sites, or adverts

Link farms Search engines value the number of incoming links on a website. Black hats link spurious websites to create the illusion of popularity for a site they are trying to boost in a search engine’s rankings

Scraper sites Parasitic web pages that cull content from other sites to draw in visitors and show them adverts

Poison pen Adding deceitful links that point towards spurious web pages or link farms. Black hats ‘poison’ people’s blog entries with the same tactic