Approximate Quality Assessment with Sampling Approaches

Data is useful to the extent that it can be quickly analyzed to reveal valuable information. With high-quality data, we can increase revenue, reduce cost, and reduce risk. On the other hand, the consequences of poor-quality data can be severe. It has been estimated that poor quality customer data costs U.S. businesses $611 billion annually in postage, printing, and staff overhead. These issues make data quality assessment a necessary and critical step in any data-related systems. Big data brings new challenges to data quality assessment due to the scale of data, streaming data, and different forms of data. Therefore, we proposed a sampling-based approximate quality assessment model on large data. Sampling large datasets can make all quality assessment processes cheaper and more feasible because of data reduction. The protocol of this work: First, the sample size is determined for estimating a large dataset. Next, sampling techniques are applied to collect samples. Then, these samples are used to estimate the quality of the large dataset. The objective of quality assessment in this work is to evaluate the completeness, accuracy, and timeliness of data and to return fast and approximate scores. Using different sample sizes and different sampling methods, we obtained 72 sets of data and compared them. These results show that the proposed approach is efficient and provides some insight into the quality assessment with samples.