Medium sized crawling made fast and easy through Lumbricus webis

Web crawlers have become popular tools for collecting large portions of the web that can be used for many tasks from statistics to structural analysis of the web. Due to the amount of data and the heterogeneity of tasks to manage, it is essential for crawlers to have a modular and distributed architecture. In this paper we describe Lumbricus webis, (short L.webis) a modular crawling infrastructure built to mine data from the .it domain and portions of the web reachable from it. The purpose of our crawler is to support gathering of advanced statistics, and advanced analytic tools on the content of the Italian Web. This paper describes the architectural features of L.webis and its performance. L.webis can currently download a mid-sized ccTLD such as “.it” in about one week.