Anti-Crawler strategy and distributed crawler based on Hadoop

With the exponential growth of network information resource, distributed Web Crawler was introduced for fetching massive web pages. However, when many websites designed a series of Anti-crawlers tactics to disturb Web Crawlers, it is our first imperative to deal with these strategies. We first elaborated on Web-Crawlers and Anti-Crawlers strategies. At the same time, this paper mainly proposed some effective measures to guarantee efficient web-crawling when faced with Anti-crawler tactics. Then we designed a experiment to analyze possible influencing factors. Finally, the future direction about distributed Web Crawels which runs on Hadoop platform was proposed by this paper.