'Cheating' the search engines

By Spencer Kelly
BBC Click

We have come to expect a lot from search engines.

Type in a phrase, and we not only expect it to find millions of relevant websites, but we also expect it to list the best or most important sites first.

Woe betide a search engine that requires me to click to page two of the results before I find the site I am looking for.

Generally they do a decent job but, up until very recently, if you were to search on the term "miserable failure", top of the Google search results was the official George Bush page on the official White House site.

Bush trick

This is an example of how even the biggest search engines can be manipulated.

Essentially, the web is a collection of pages, all linking to each other.

If you are searching for something, search engines like Google, Ask, and Yahoo! will first find all the pages they think are relevant.

Then, crucially, they need to decide which order to display search results in. One of the most important factors in deciding how relevant particular sites are is to count how many other sites link to it.

The more references, or links there are to a site, the more important it is deemed to be.

Because lots of people quote and link the BBC website, for example, the BBC site is seen as relevant and a good hit. That is why the site ranks quite highly for many search terms.

An underground movement of bloggers exploited this fact, to create a "link bomb". They encouraged thousands of their peers to include a link to the Bush homepage in their blogs, and label it "miserable failure".

With all of these links, the main search engine algorithms were fooled into thinking this was a relevant result for that search term, and Bush was driven to the top of the rankings. Mr Bush had been link bombed.

At first, the search engines ignored it - it was not their job to censor search results, however controversial.

Nor did they censor other link bombs - googling "liar" would show Tony Blair's homepage first.

Link bombs are usually self-defeating - as they become successful, other popular sites begin to discuss the bombs, and end up becoming more popular than the link bomb, driving it from number one.

Apostolos Gerasoulis, co-inventor of search technology for Ask.com said: "I don't think this is the problem though. This is just fun."

"Why we don't remove it, or why the search engines don't remove it, is because as long as we give relevant results for most of the queries that you type then we're OK.

"The impact of this bombing, as it is called, is minimal, insignificant. It might be two or three examples," he added.

Google decided to tweak its search algorithm to spot link bombs, and the miserable failure dropped away. On other search engines, such as Ask, it remains high.

If they build the right page, it is OK, if they build a great page it's OK but the problem comes when they really try to cheat you
Apostolos Gerasoulis search expert

The link bomb actually achieved what any business would love to have - the number one search result in their category.

The ideal place to be is in the centre of the search result, the so-called natural search result, online marketer Fadi Shuman, explained.

"You get 80% of the clicks from the natural results. That is where we all want to be. The way to achieve that is through search engine optimisation."

Optimisation is produced by making the website as visible to search engines as possible and having other sites linking to it.

However search engine optimisation also has a darker side.

Blacklisted

In February 2006, BMW Germany got into hot water over its website.

The problem was that the site was laden with keywords which customers could not see, but which search engines could.

They were there specifically to boost the rankings. Google has rules to guard against so-called black hat methods - specifically, "don't present different content to search engines than you do to your users".

Once discovered, Google blacklisted the site and dropped it from their search results altogether until BMW redesigned the site.

Jason Duke, search engine optimiser, said: "We try and persuade people to link to us, whether that be via content, whether that be via methods of creating controversy in the marketplace."

It is an ongoing battle of sorts, between the search engines and the optimisers who would love to work the system to their clients' advantage. Although both sides prefer to call it an uneasy truce.

Mr Gerasoulis added: "If they build the right page, it's OK, if they build a great page, it's OK. But the problem comes when they really try to cheat you."

Story from BBC NEWS:
http://news.bbc.co.uk/go/pr/fr/-/2/hi/programmes/click_online/6526393.stm

Published: 2007/04/05 14:04:17 GMT

© BBC MMVII