Comprehensive Functional ETL Testing Methodologies for Real-World Data

In today's data-driven enterprises, data warehouses are crucial for aggregating diverse datasets for analysis and research. The ETL (Extract, Transform, Load) process is central to this, involving data extraction, integration, cleansing, and loading into the warehouse. Challenges in ETL can cause inaccuracies, diminishing the value of high-quality source data. Robust ETL testing methodologies are essential to ensure correct implementation and efficient execution. This paper surveys existing functional ETL testing approaches, categorizing them into data quality and balancing tests. Data quality tests identify issues like missing values and outliers in the target data, while balancing tests verify data consistency between source and target. The research highlights unresolved challenges in completeness and scalability, proposes future research directions, and includes an empirical study of ETL testing on the Our World in Data warehouse, identifying data disparities and abnormalities.