A Stable Algorithm Based on Pareto Optimal and Maximum Difference for Record Matching with Errors

Mining information from different organizations is beneficial for many big-data applications related to providing better services. Record matching is a fundamental task for analysing multisource datasets. However, due to data privacy, sensitive information of the data(e.g., ID, name, or social security number) is usually removed before the data can be shared. Without such identification information, one has to calculate similarity between data records as references for matching records across datasets. Unavoidably, this will introduce errors in record matching because the calculation of similarity score is based on imperfect information. This paper proposes a two stages record matching method-Pareto Optimal and Maximum Difference Algorithm(POMDA). The POMDA considers the dominance similarity scores in each attribute and the maximum difference value of each record pair similarity score. In addition, this paper introduces thresholding and perturbation technique to adjust similarity scores of records to improve matching accuracy. Experimental results show that the proposed PMODA method with thresholding technique is more stable and accurate than greedy method(GM) and Hungarian algorithm(HA). Experiment with synthetic datasets show that three record matching method with thresholding technique will get better matching result in highly correlated dataset.