Approximate Query Processing Using Wavelets in OLAP with Arbitrarily Sized Data and Bounded Errors

Analysis of the existing techniques for approximate query processing of Big Data, based on sampling, histograms and wavelets, demonstrates that wavelet-based methods can be effectively utilized for OLAP purposes due to their advantages in terms of handling multidimensional data and querying single cells as well as aggregate values from a data warehouse. At the same time the current wavelet-based methods for approximate query processing have certain deficiencies making difficult to implement them in practice. In particular, most of the techniques struggle with arbitrarily size data either imposing a restriction on a dimension length to be a multiple of a power of two, or complicating decomposition algorithms what leads to the construction time increase and difficulties with error estimations. Also, there is a lack of methods for approximate processing based on wavelets with a bounded error and a confidence interval for both single and aggregate values. Our contribution in this paper is introduction of a new wavelet method for approximate query processing which handles arbitrarily sized multidimensional datasets with minor extra calculations and provides a bounded error of the single or aggregate value reconstruction. It is demonstrated that the new method allows evaluating a confidence interval of the query error depending on a given compression ratio of a data warehouse or performing an inverse task, i.e. evaluating the required data warehouse compression ratio for a given allowable error. The introduced method was applied and verified over real epidemiological datasets to support research in finding correlations and patterns in disease spread and clinical signs correlations. It was demonstrated that the accuracy of the estimated error is acceptable for retrieving single and aggregate value, query time processing advantage depends on compression ratio and volume of the processed data.