A High Performance Compression Method for Climate Data

Climate modeling data are usually multidimensional arrays of floating-point numbers. These arrays typically have two or three spatial dimensions and one temporal dimension, describing the evolvement of climate variables in a time span. With the advances of high performance computing, the volume of climate data is expanding exponentially, bringing tough challenges for climate data archiving and sharing. In this paper, we propose a lossless compression algorithm for the time-spatial climate floating-point arrays. Our compression algorithm can eliminate more data redundancy efficiently through adaptive prediction, XOR-differencing, and multi-way compression. In addition, static regions, which are very common in climate data, can be identified and compressed more efficiently. Moreover, to utilize the multi-cores on modern computers, we proposed a method to parallelize our compression algorithm. Evaluations demonstrate that single thread version of our compression method can achieve the best balance in compression ratios, deflating throughputs and inflating throughputs. And the parallel version can achieve 800 MB/s deflating throughputs and over 2600 MB/s inflating throughputs on a 16-core server.