Microsoft to clean up web spam

Researchers at Microsoft have developed a tool to scrub search engines of major websites that pollute search results and ultimately help clean the web of spam.

Called Strider Search Defender, the tool is designed to dig out web pages that are a front for spam sites, according to a paper published by Microsoft researchers on Thursday. These web pages typically reside on blogging sites and other services that provide free web space, the researchers said.

Spammers soil the web with countless links to their spam fronts in order to gain a higher ranking in search engines. Yi-Min Wang, principal researcher at Microsoft, said in an interview: "By cleaning up web search, hopefully we can discourage spammers from cluttering the web with spam."

Microsoft's tool doesn't find spam the traditional way, by looking at the site's content. Instead, it turns the spammers' activities against them by using search engines to find links to potential spam pages. These links are often posted as comments on blogs, in online discussion forums and in guestbooks, also called "comment spam".

Search Defender starts with a list of confirmed spam web addresses. A "Spam Hunter" part of the tool runs those addresses through search engines to find pages that link to the spam sites, using the "link:" query tag. Additional spam URLs found on those sites are, in turn, run through the Spam Hunter, resulting in a long list of potential spam sites.

Then, using another Microsoft research project, Strider URL Tracer, false positives are filtered out and a list of web pages that redirect to spam sites is compiled. Strider URL Tracer actually visits each one of the web addresses found by the Spam Hunter to see if it redirects to a secondary spam page.

Wang said: "We use search engines to find them. Spammers are basically telling us: here are my spam URLs."

Spammers use various online services to host spam fronts, including free web hosting providers such as Angelfire, Tripod and Yahoo!'s Geocities, Microsoft said. Blogging services are also often abused, Google's Blogger at blogspot.com is especially popular, according to the research report.

The report said: "Our preliminary investigation shows that spam blogs hosted on blogspot.com appear to be particularly widely spammed and effective against search engines."

Microsoft's researchers are working with the MSN Search team to see how the search service could be cleaned up, Wang said. Additionally, he called on the web community, especially the operators of blog and free hosting sites, to co-operate to combat the web spam problem.

Wang added: "In the end it is all about protecting the search engines. Because if the spam doesn't show up in any search engine result the spammer will not receive traffic."

Joris Evers writes for CNET News.com