Scheduled System Maintenance:
On May 6th, system maintenance will take place from 8:00 AM - 12:00 PM ET (12:00 - 16:00 UTC). During this time, there may be intermittent impact on performance. We apologize for the inconvenience.
By Topic

Near-Duplicates Detection for Vietnamese Documents in Large Database

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

The purchase and pricing options are temporarily unavailable. Please try again later.
3 Author(s)
Cong Thanh Truong ; Vietnam Nat. Univ., Hanoi ; The Duy Bui ; Bao Son Pham

Near-duplicate documents exacerbate the problem of information overload. Research in detecting near-duplicates has attracted a lot of attention from both industry and academia. In this paper, we focus on addressing this problem for Vietnamese documents which, to the best of our knowledge, has not been done before. Most of the current algorithms have been designed for English which are not directly applicable to Vietnamese - a monosyllabic language. We propose to combine Charikarpsilas algorithm (Alexander Andoni et al., 2006) with a ldquoweighting schemerdquo and Vietnamese specific features to address the language intricacy. Experimental results indicate that our scheme is effective for detecting near-duplicates in a corpus of Vietnamese documents.

Published in:

Advanced Language Processing and Web Information Technology, 2008. ALPIT '08. International Conference on

Date of Conference:

23-25 July 2008