Classifier based duplicate record elimination for query results from web databases

Record matching is an essential step in duplicate detection as it identifies records representing same real-world entity. Supervised record matching methods require users to provide training data and therefore cannot be applied for web databases where query results are generated on-the-fly. To overcome the problem, a new record matching method named Unsupervised Duplicate Elimination (UDE) is proposed for identifying and eliminating duplicates among records in dynamic query results. The idea of this paper is to adjust the weights of record fields in calculating similarities among records. Three classifiers namely weight component similarity summing classifier, support vector machine classifier and one class support vector machine classifier are iteratively employed with UDE where the first classifier utilizes the weights set to match records from different data sources. With the matched records as positive dataset and non duplicate records as negative set, the second classifier identifies new duplicates. Then, one-class support vector machine classifier is employed for further detecting the duplicates. The iteration stops when no duplicates can be identified. Thus, this paper takes advantage of dissimilarity among records from web databases and solves the online duplicate detection problem.