A Study on the Impact of Missing Values in Probabilistic Matching

Deterministic matching and probabilistic matching are two Entity Resolution (ER) methodologies which are widely used in the industry. Anecdotally, probabilistic matching has a reputation for being sensitive to certain data quality problems, particularly missing values and inconsistent representation of values. This paper explores the impact of missing values on probabilistic matching by comparing three methods for the treatment of missing values. The experiment has two independent variables. The first variable is the number of missing values for a particular attribute. The second is the treatment of the missing values in the calculation of the scoring weights. The results show that for the conditions of the experiment, probabilistic matching did not exhibit any higher degree of sensitivity to missing values than deterministic matching. Secondly, it demonstrated that the navïe treatment of assigning a weight of zero (0) to missing value agreements resulted in much poorer performance that either the baseline treatment of ignoring missing values or treatment of calculating a probabilistic weight for missing values.