Webpage Duplicate Detection Using Combined POS and Sequence Alignment Algorithm

Combined syntactical categories and sequence alignment algorithms are implemented and used to weed-out duplicate and near-duplicate Web-pages from search engine results. The syntactical structures manifested as POS-tags were pre-processed using a POS tagger converting parts of a Webpage's text into a string of tags. The produced string was then subjected into the longest common sequence (LCS) techniques (as is commonly done in computational biology), to detect duplicate and near-duplicate Webpages. The process of tagging and aligning was based on set of sentences extracted from the Web page as a representative of the pages. The query-keywords are used as a basis for sentence extraction. Results obtained from experiments performed have shown that such a combined approach can provide very interesting similarity calculation and re-ranking measure. This can be used with reasonable efficiency to detect duplications on search results generated by search engines such as Google. Similarity measurements obtained can be further used as a basis for text analysis of the search results allowing the detection of duplicate and near duplicates and clustering of documents in general.