AuToCrawler: an integrated system for automatic topical crawler

A topical (or focused) crawler is a Web crawler aiming to search and retrieve Web pages from the World Wide Web, which are related to a specific topic. Rather than downloading all accessible Web pages, a topical crawler analyzes the frontier of the crawled region to visit only the portion of the Web that contains relevant Web pages, and at the same time, try to skip irrelevant regions. This leads to significant savings in both computation and communication resources. In this paper, we present an integrated topical crawler: AuToCrawler. The main features of AuToCrawler consist of a user interest specification module that mediates between users and search engines to identify target examples and keywords that together specify the topic of their interest, and a URL ordering strategy that combines features of several previous approaches and achieves significant improvement. It also provides a graphic user interface such that users can evaluate and visualize the crawling results that can be used as feedback to reconfigure the crawler.