The Design and Implementation of a Spider in Local Network

This paper mainly introduces the principle and method of the search engine, it also gives the design and implementation of the multi-thread concurrent spider based on the local network. This spider adopts the BloomFilter to solve the URL duplicate and thread pool to manage the concurrent threads; it uses the IoC technique in Spring to provide the support of the different file formats such as DOC, PDF, XLS etc which can demonstrate the scalability of the whole application; the spider speeds up the I/O performance by storing the data in the light database. At the end of the paper, we give the comparison and the analysis between the local search engine and general business search engine in the efficiency and performance.