Adaptive Per-File Lossless Compression of Floating-Point Data

The large amount of floating-point data generated by scientific applications makes data compression essential for I/O performance and efficient storage. However, floating-point data is difficult to compress losslessly, and most compression algorithms are only effective on some files. In this paper, we study the benefit of compressing each file with a potentially different algorithm. For this purpose, we created AdaptiveFC, which is based on a tool that can chain data transformations together to generate millions of compression algorithms. AdaptiveFC uses a genetic algorithm to quickly identify an effective compressor in this vast search space for a given file. A comparison of AdaptiveFC to 15 leading lossless GPU compressors on 77 files from 6 datasets in the SDRBench suite shows that per-file compression yields higher compression ratios on average than any individual algorithm.