Scoring Matrix for Unstandardized Data in Entity Resolution

The research describes how a "scoring matrix" comparator was used in an entity resolution (ER) process to allowing entity reference sources to be ingested and compared without pre-processing. In this method, each entity reference is tokenized and all pairs of tokens between the references are scored by a similarity scoring function such as the Levenshtein edit distance. The pairwise scores are normalized into an aggregate score which is then compared to a threshold value to determine ER link, not link, and review decisions. The research describes a particular design of the scoring matrix comparator as implemented in the OYSTER open source ER system. It also provides a comparison of precision and recall performance of the scoring matrix against traditional Boolean ER logic using benchmark datasets. The research described here shows how using even a relatively simple scoring matrix can produce results comparable to standard techniques, but without the need for pre-processing reference heterogeneous standardized formation into single standard.