Data Cleansing Methods for Big Data: A Systematic Review

Given the rapid growth in the amount of data available, and the challenges posed by its inherent inconsistency and the integration of multiple data sources, the need for effective data cleansing techniques has become increasingly salient. This review aims to provide a comprehensive overview of recent developments in data cleansing approaches for big datasets. Firstly, this paper provides an overview of the characteristics of big data based on the common five V‒model, as well as the importance of data quality by presenting the most common anomalies in such datasets. Additionally, the need for big data specific cleansing methods is addressed. As part of the systematic literature review, well‒known scientific databases have been searched through for recent publications focusing on big data cleansing. Following a thorough review of the extant literature, the approaches have been categorized to highlight the research domains employed in big data cleansing. The results underscore the necessity for a holistic approach, facilitating end‒to‒end data cleansing. Furthermore, the proportion of publications incorporating machine learning or artificial intelligence into data cleansing techniques underscores the need for further research in this area.