The research and implementation of Web Spider in Search Engine

With the further research for Web Spider in Search Engine, innovative design of the original web page storage format is presented to achieve both savings in physical storage space and increase the efficiency of procedures for the operation of I/O; In order to distinguish different web pages and evaluate the importance of data in webpages, a definition of UrlRank algorithm is presented; Storing data with uniform coding, designing a code conversion algorithm, which convert data with different types to an uniform format; Using parallel multi-threaded Spider technology to design and implement web spider system, which system improves the efficiency of the climbing page, and support the continued climb breakpoint, through the process of testing, we can verify the effectiveness and integrity of the program function.