Sensitivity and Impacts on Parallel Compression of Prediction of Lossy Compression Ratios for Scientific Data

Compression ratio estimation is an important optimization of I/O workflows processing terabytes of data. Applications such as compression auto-tuning or lossy compressor selection require a high-throughput, accurate estimation to be fast. Prior works that utilize sampling are fast but inaccurate, while approaches that do not use sampling are accurate but slow. We present a novel, lightweight sampling technique that excels in speed and accuracy, by leveraging both statistical and spatial properties of dataset samples. Through a comprehensive sensitivity analysis we show that there is no one-size-fits-all sampling strategy because of differences in compression principles, and accounting for these differences has a direct impact on overall prediction accuracy in the methods that use these predictions. Our estimation technique demonstrates superior ($1.36 \times$ to $11 \times$) prediction accuracy compared to existing sampling methods while significantly improving data throughput by $3.46 \times$ compared to non-sampling approaches.