By Topic

Webpage Duplicate Detection Using Combined POS and Sequence Alignment Algorithm

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)
Elhadi, M. ; Dept. of Comput. Sci., Sultan Qaboos Univ., Oman ; Al-Tobi, A.

Combined syntactical categories and sequence alignment algorithms are implemented and used to weed-out duplicate and near-duplicate Web-pages from search engine results. The syntactical structures manifested as POS-tags were pre-processed using a POS tagger converting parts of a Webpage's text into a string of tags. The produced string was then subjected into the longest common sequence (LCS) techniques (as is commonly done in computational biology), to detect duplicate and near-duplicate Webpages. The process of tagging and aligning was based on set of sentences extracted from the Web page as a representative of the pages. The query-keywords are used as a basis for sentence extraction. Results obtained from experiments performed have shown that such a combined approach can provide very interesting similarity calculation and re-ranking measure. This can be used with reasonable efficiency to detect duplications on search results generated by search engines such as Google. Similarity measurements obtained can be further used as a basis for text analysis of the search results allowing the detection of duplicate and near duplicates and clustering of documents in general.

Published in:

Computer Science and Information Engineering, 2009 WRI World Congress on  (Volume:1 )

Date of Conference:

March 31 2009-April 2 2009