An incremental clustering crawler for community-limited search

We propose an incremental clustering crawler, a novel algorithm for finding communities for community-limited search in the web. A web community is a set of semantically related sites found through link-based clustering. The key idea of the proposed algorithm is to perform clustering incrementally while crawling is in progress. This algorithm does not need to crawl all the web pages a priori, but needs to crawl only as many web pages as are relevant to the clusters that are being formed. This ability to crawl on the fly is an important advantage since it is infeasible to crawl the entire set of web pages in the world and since we often do not even know which web pages or sites to crawl. Another advantage is that the time spent on clustering is reduced because at any time the clustering is performed on only the relevant web pages collected thus far. An apparent disadvantage is that the resulting clusters are not optimal since the algorithm does not have all the crawled sites available at the time of clustering. Experiments show, however, that the achieved cluster quality is comparable to the optimal cluster quality which, in our experiments, is achieved using the minimum spanning tree clustering algorithm.