Evolutionary study of web spam: Webb Spam Corpus 2011 versus Webb Spam Corpus 2006

With over 2.5 hours a day spent browsing websites online [1] and with over a billion pages [2], identifying and detecting web spam is an important problem. Although large corpora of legitimate web pages are available to researchers, the same cannot be said about web spam or spam web pages. We introduce the Webb Spam Corpus 2011 - a corpus of approximately 330,000 spam web pages - which we make available to researchers in the fight against spam. By having a standard corpus available, researchers can collaborate better on developing and reporting results of spam filtering techniques. The corpus contains web pages crawled from links found in over 6.3 million spam emails. We analyze multiple aspects of this corpus including redirection, HTTP headers and web page content. We also provide insights into changes in web spam since the last Webb Spam Corpus was released in 2006. These insights include: 1) spammers manipulate social media in spreading spam; 2) HTTP headers also change over time (e.g. hosting 'P addresses of web spam appear in more 'P ranges); 3) Web spam content has evolved but the majority of content is still scam.