From data integrity to inference integrity

Data integrity was recognized as crucial for databases since in banking and accounting, the earliest important application domains, errors were intolerable owing to legal ramifications; the painfully labor intensive process of the error recovery was another disincentive. Semantic Integrity constraints, which were articulated by the designer through a study of the domain, provided rules; they prevented a great deal of errors. The advent of data mining led to a reversal: data was used for the inference of rules: ‘interesting’ association patterns. The users, who were analysts, were warned that these were merely correlations and not causations. More recently, we have begun to witness how, relying on data science, unreliable data is being subjected to largely opaque inference algorithms, typically based on machine learning, with tremendous consequences for human lives. Our challenge is to enhance the integrity of the inference process.