Data mining for removing fuzzy duplicates using fuzzy inference

Data cleaning deals with the detection and elimination of inconsistencies in data, gathered from distributed sources. This process is essential for drawing correct conclusions from data, in decision support systems. Mining the data for the removal of fuzzy duplicate records is a challenging part the cleaning process. The vagueness and uncertainty involved in the detection of fuzzy duplicates make it a niche for applying fuzzy reasoning. Although, uncertainty algebras like fuzzy logic are known, their applicability to the problem of duplicate elimination has not been explored. In this paper, a practical and novel duplicate elimination system is presented, which exploits a fuzzy inference engine for handling the uncertainty involved in detecting fuzzy duplicates. The innovation of the system is in capturing expert's knowledge, in the form of natural language fuzzy rules and using these simple rules to efficiently clean the data. This in turn, reduces the time required for the repetitive and time consuming task of hard-coding, for de-duplication based on a schema for each database.