The Research of Web Page De-duplication Based on Web Pages Reshipment Statement

Web page de-duplication module is an important part of search engine system, which can improve its performance and quality with filtering the web pages downloaded by crawler system of search engine and eliminating the duplicated web pages. This paper from the source of duplicated web pages--reshipment proposes a web page de-duplication method that the information including original websites and web titles are extracted to eliminate duplicated web pages based on feature codes. Experiments show that this method can achieve satisfactory results in eliminating large-scale duplicated web pages.