Comparing Apples and Oranges: Comparability problems caused by intransparent Data Cleansing for AI Applications

No AI applications without data. Data is like oil for combustion engines - the most important “fuel” for AI applications. That’s why great attention must be paid to data provision and data quality assurance. Real-world data usually contains fuzziness, errors, outliers, and often many unusable records. There are myriad reasons why data in its original form is often unsuitable for model development (e.g., technical errors, unexpected user input). Data cleaning is therefore an extremely critical and tedious task that forms the basis for high-quality results with AI algorithms. Although this fact is widely known, the importance of this step is usually not considered in depth when comparing different AI algorithms and applications, leading to misleading comparisons. In this paper, we therefore provide an overview of the most common challenges in data cleaning to make AI comparable. To help address these challenges, we propose a novel systematic and machine-readable data cleaning protocol to improve comparability and reproducibility in the field of AI.