Progressive Entity Matching via Cost Benefit Analysis

Entity matching (EM) is a fundamental problem in data preprocessing, and is a long running topic in big data analytics and mining communities. In big data era, (nearly) real-time data applications become popular, and call for progressive EM, which produces as many match pairs as possible in very limited time. Previous progressive EM focus on memory based solutions, but disk based solutions are necessary when dirty datasets cannot be fully loaded into memory. To this end, we propose a cost benefit analysis based progressive EM approach, which partitions data according to coarse clustering results and then iteratively schedules data partitions in a greedy way for high progressive resolution. At first, based on estimated record pair similarities, records are fast coarsely clustered; then, record clusters with near average similarities are greedily distributed to the same partitions, and data partitions are cached. After that, cost model is defined with time and space constrains, and benefit model is defined with expected resolution results. On the basis of the cost benefit model, a greedy approximate method is proposed to effectively schedule data for high progressiveness of EM. Finally, we implement extensive experiments over several datasets to evaluate our approach, and show its advantages over existing works.