The Research and Implementation of Parallel Web Crawler in Cluster

As the foundational component of web information acquisition, web crawler has been always the research hotspot in academia and industry, recently the parallel web crawler is the main research direction. In view of the shortage of the center-like dynamic assignment and distributed static assignment which are adopted by current parallel web crawler, this paper presents a parallel web crawler based on the cluster environment, which adopts the dynamic assignment structure, introduces the distributed controller pattern, eliminates the problem of single point failure in the central controller, and enhances system's concurrent capability. In addition, it uses the URL dynamic assignment technology, and realizes dynamic balance among components according to their real-time condition.