Toward a Novel Measurement Framework for Big Data (MEGA)

Big Data is quickly becoming a chief part of the decision-making process in both industry and academia. As more and more institutions begin relying on Big Data to make strategic decisions, the quality of the underlying data comes into question. The quality of Big Data isn’t always transparent and large-scale systems may even lack its visibility, which adversely affects the credibility of the Big Data systems. Continuous monitoring and measurement of data quality is therefore paramount in assessing whether the information can serve its purpose in a particular context (such as Big Data analytics, for example). This research addresses the need for Big Data quality measurement modeling and automation by proposing a novel conceptual quality measurement framework for Big Data (MEGA) with the purpose of assessing the underlying quality characteristics of Big Data (also known as the V’s of Big Data) at each step of the Big Data Pipelines. The theoretical quality measurement models for four of the Big Data V’s (Volume, Variety, Velocity, Veracity) are currently automated; the remaining 6 V’s (Vincularity, Validity, Value, Volatility, Valence and Vitality) will be tackled in our future work. The approach is illustrated on a case study.