A Hybrid Data Cleaning Framework Using Markov Logic Networks

With the increase of dirty data, data cleaning turns into a crux of data analysis. The accuracy limitation of the existing integrity constraints-based cleaning approaches results from insufficient rules. In this paper, we present a novel hybrid data cleaning framework on top of Markov logic networks (MLNs), termed as
${\sf MLNClean}$
, which is capable of learning instantiated rules to supplement the insufficient integrity constraints.
${\sf MLNClean}$
consists of two steps, i.e.,
pre-processing
and
two-stage data cleaning
. In the pre-processing step,
${\sf MLNClean}$
first infers a set of probable instantiated rules according to MLNs and then builds a two-layer MLN index structure to generate multiple data versions and facilitate the cleaning process. In the two-stage data cleaning step,
${\sf MLNClean}$
first presents a concept of
reliability score
to clean errors within each data version separately, and afterward eliminates the conflict values among different data version using a novel concept of
fusion score
. Considerable experimental results on both real and synthetic scenarios demonstrate the effectiveness of
${\sf MLNClean}$
in practice.