Assessing Context-Aware Data Consistency

Data analysis is a demanding task that involves extracting deep insights hidden in data. Many businesses enforce data analysis irrespective of the domain, as it is crucial in minimizing developmental risks. Raw data cannot be used to perform any analysis as poor-quality data leads to erroneous decision-making. This makes data quality assessment a necessary function before data analysis. Data quality is a multi-dimensional factor that affects the analysis in multiple ways. Among all the dimensions, consistency is one of the most critical dimensions to assess. Context of data plays an important role in consistency assessment, as the records are inherently related within a dataset. Existing studies are computationally expensive and do not consider the context of data. In this paper, we propose a comprehensive context-aware data consistency assessment tool that uses machine learning to evaluate the consistency of data. Our model was developed on Apache Hadoop and Apache Spark to support big data, as well as to boost some computationally intensive algorithms.