Performance Improvement on k²-Raster Compact Data Structure for Hyperspectral Scenes

This letter proposes methods to improve data size and access time for <inline-formula> <tex-math notation="LaTeX">$k^{2}$ </tex-math></inline-formula>-raster, a losslessly compressed data structure that provides efficient storage and real-time processing. Hyperspectral scenes from real missions are used as our testing data. In previous studies, with <inline-formula> <tex-math notation="LaTeX">$k^{2}$ </tex-math></inline-formula>-raster, the size of the hyperspectral data was reduced by up to 52% compared with the uncompressed data. In this letter, we continue to explore novel ways of further reducing the data size and access time. First, we examine the possibility of using the raster matrix of hyperspectral data without any padding (unpadded matrix) while still being able to compress the structure and access the data. Second, we examine some integer encoders, more specifically the Simple family. We discuss their ability to provide random element access and compare them with directly addressable codes (DACs), the integer encoder used in the original description for <inline-formula> <tex-math notation="LaTeX">$k^{2}$ </tex-math></inline-formula>-raster. Experiments show that the use of unpadded matrices has improved the storage size up to 6% while the use of a different integer encoder reduces the storage size up to 6% and element access time up to 20%.


I. INTRODUCTION
H YPERSPECTRAL scenes are data taken from the air by sensors, such as airborne visible/infrared imaging spectrometer (AVIRIS), or from space by satellite instruments such as Hyperion and infrared atmospheric sounding interferometer (IASI). These scenes are made up of multiple bands from across the electromagnetic spectrum, and data extracted from certain bands have many practical applications, such as oil field exploration and mineral exploration. Due to their relatively large sizes, hyperspectral scenes are usually compressed to increase transmission throughput and reduce data volumes.
Compact data structures can store data efficiently and provide real-time data compression and access to the original data [1]. They can be loaded into main memory, and operations to access data are often carried out by means of the rank and select functions [2]. Compact data structures provide reduced space usage and query time. There is no need to decompress a large portion of the structure to access and query individual data as it is the case with data compressed by classical compression algorithms such as gzip and specialized algorithms such as CCSDS 123.0-B-2 [3].
In this work, we reduce the hyperspectral data size using k 2 -raster, a compact data structure, to produce lossless compression. In our previous letter [4], we presented a predictive method and a differential method that made use of spatial and spectral correlations in hyperspectral data with favorable results. Nevertheless, due to the nature of these methods, only random access to individual cells can be done, whereas other operations such as query on a region cannot be performed. In this letter, we focus on investigating whether unpadded matrices and variable-length integer encoders other than directly addressable codes (DACs) [5] can provide competitive compression ratios as well while improving random and query access time. In our case, we need to store non-negative small integers in the k 2 -raster tree structure, which is built in such a way that the nodes are not connected by pointers but can still be reached with the use of a compact data structure's linear rank function. Fig. 1 depicts a global picture of the interrelations between the elements discussed above. The compact data structures that we have been working on are still at the research stage when applied to hyperspectral scenes, but with the encouraging results that we have obtained so far, we can extrapolate their practical use in applications of remote sensing and geographic information systems [6]- [10].
The letter is organized as follows. Section II provides background information on k 2 -raster built from a padded matrix and the various integer encoders. Section III describes the proposed method of using an unpadded matrix to build the structure and introduces the different variable-length integer encoders. Section IV presents some experimental results. Section V sums up the key points of our discussion.
II. BACKGROUND Ladra et al. [11] proposed k 2 -raster, a tree structure specifically designed for raster data including images. It is built from a matrix of width w and height h, and an integer k ≥ 2. If the matrix can be partitioned into k 2 square quadrants of equal size, it can be used directly. Otherwise, it is necessary to enlarge the matrix to size s × s, where s = k log k max(w, h) , and the number of subdivisions is log k (s). The padding elements are equal to zero. This extended (padded) matrix is then recursively partitioned into k 2 square submatrices of identical size, hereafter referred to as quadrants. This process is repeated Fig. 1. Construction of k 2 -raster. A k 2 -ary tree is first built from a raster matrix. Compression and random access are achieved when tree node data are encoded by an integer encoder, such as DACs, Simple-9, Simple-16, or PForDelta, resulting in a k 2 -raster structure. until all cells in a quadrant have the same value, or until the submatrix has size 1 × 1 and cannot be further subdivided. This partitioning produces the nodes for a k 2 -ary tree topology where the data in the nodes are stored in the following data structures. 1) At each tree level , the maximum and minimum values of each quadrant are computed. These are then compared with the corresponding maximum and minimum values in the parent node, and the differences are stored in the V max and V min arrays of each level. Saving the differences instead of the original values results in smaller values for each node, which in turn allows a better compression with an integer encoder. Next, with the exception of the root node at the top level, the V max and V min arrays at all the levels are concatenated to form L max and L min , respectively. Both arrays are then compressed by an integer encoder.
2) The root's maximum (r Max) and minimum (r Min) values are stored as uncompressed integers.

3) A bitmap array T is generated from all the nodes except
the ones at the root and at the last level, each node indicating whether it has child nodes or not. This bitmap serves to locate the tree nodes when cell queries are performed by means of a rank function [2].
In Fig. 2, an example of a 5×5 matrix is shown to illustrate this process. The elements which fully describe the resulting k 2 -raster structure are shown at the bottom of Fig. 2. Please refer to [12] for a more comprehensive description of k 2 -raster.
DACs were proposed by Brisaboa et al. [5]. Consider a sequence of integers x. Each integer x i , which is represented by log 2 x i + 1 bits, is broken into chunks of bits of size C S . Each chunk is stored in a block of size C S + 1 with the additional (highest) bit used as a control bit (0 for most significant bits, 1 otherwise). More information on DACs and software implementation can be found in the paper by Brisaboa et al. [5].
Simple-9 word-aligned encoding [13] is another approach to compression. Each 32-bit word is split into two parts: a 28-bit part where a variable number of integers are encoded and a 4-bit part which is a selector with a value ranging from 0 to 8. For example, selector 0 signals that the first 28 integers in the data have a value of 0 or 1. Selector 1 signals that it can pack 14 integers into the segment with a maximum bit length of 2 bits for each. Beginning with selector 0, each selector is tested until the most efficient one is found. Simple-16 [14] is a variant of Simple-9 and uses all the 16 combinations in the selector bits. Their values range from 0 to 15. PForDelta [15] is also similar to both Simple-9 and Simple-16 but encodes a fixed group of 32, 64, 128, or 256 integers in a varying number of bytes. A predetermined percentage of those numbers that are larger than the others are encoded separately after the smaller numbers or in another location.

A. Proposed Building k 2 -Raster Without Padding
As mentioned in Section II, given a k value, a matrix with a size that is not a power of k 2 needs to be extended according to the equation for computing s in that section. The values of the new cells are set to zero. This is necessary because the search from the root down to its leaves requires the knowledge of its child node location using the rank function, which is a function of the number of child nodes each parent node has. Adding new cells, however, also means that the nodes that are outside the matrix have to be saved, and this leads to a larger structure.
To illustrate the above point, we construct a tree based on the elements within the bounds of the original matrix where k = 2. This matrix together with its corresponding k 2 -raster tree are shown in Fig. 3. This is done without padding and is therefore not a full quaternary tree (a full quaternary tree is one where each node has either 0 or 4 child nodes). In this case, the parent does not know how many children it has, so it is not possible to use the rank function to get to the correct child nodes without including them in the T bitmap. On the other hand, with padding, as explained earlier in Section II, it is a full quaternary tree. Fig. 2 shows the padded matrix and the tree that is built from it. Comparing it with Fig. 3, we can see that the k 2 -raster tree from the unpadded matrix can save fewer elements.
To build a non-full quaternary tree, we modify the original function for building the tree and prune the values that are located outside the bounds of the original matrix. This reduces the number of L max and L min values that need to be saved. However, when forming the T bitmap, the full-quaternary tree is still used to ensure that the parent nodes can reach their child nodes, with the corresponding bit value of the node that is outside the bounds being set to zero. This facilitates the search path through the full quaternary tree when cell queries are performed using the rank function. Note that the T bitmap does not have to store the location information of the nodes of the last level because we can compute the location of the values from the original matrix size and the s value in the expanded matrix.
The algorithms for building k 2 -raster with and without padding are listed in Algorithm 1. What this build function does is it excludes elements that are outside the bounds of the original matrix and save only the actual data. This helps reduce the size of the structure. To compute the theoretical storage reduction, we can count how many symbols 0 we are sparing in the encoding with the unpadded matrix: spared0. Since the actual reduction in storage depends on the entropy encoding, we could estimate the saving to be spared0 × log 2 (Probability(spared0)). Another way to estimate space saving is to calculate the size of the original raster matrix originalDim compared with that of the expanded matrix expandedDim. The maximum saving can be estimated to be ((expandedDim − originalDim)/expandedDim)×100% but the saving is, in general, less, due to factors such as the k value, tree height, and k 2 -raster saving such as nodes at a higher level that become leaves. For example, in Figs. 2 and 3, the estimated maximum saving is ((64 − 25)/64) × 100% = 61% and the L max nodes' saving is actually ((56 − 34)/56) × 100% = 39%. Hence, there is a relationship between the image dimensions and the storage saving.

B. Random Access for Integer Encoders
In saving the L max and L min arrays, the authors of k 2 -raster made use of DACs as an integer encoder for random access to its elements. In this research, we have investigated other word-aligned integer encoders from the Simple family: Simple-9 and Simple-16, and the PForDelta variant, which also allow random access due to their structure.

Algorithm 1:
Algorithm for Constructing T, V max, V min for a Padded Matrix and an Unpadded Matrix. With the "+" Lines Removed, the Pseudocode Is for the Function Build(n, nx, ny, l, r, c, o l ) for Building From an Unpadded Matrix M. With the "−" Lines Removed From the Code and the "+" Lines Re-Added, It Becomes One for the Function Build(n, l, r, c (childmax, childmi n, childoutcount) ← Build(n/k, nx, ny, l + 1, r + i · (n/k), c + j · (n/k), o l ) pmi n l ← pmi n l + 1 maxval ← max(maxval, childmax) mi nval ← mi n(mi nval, childmi n) + if maxval = mi nval then + pmax l ← pmax l − k 2 + pmi n l ← pmi n l − k 2 + return (maxval, mi nval) -return (maxval, mi nval, outcount) The use of partial sums and sampling described in [1, §3.3 and §4.2] can be used in the Simple family of codes to expedite the search process in the compressed array. We incorporate such strategies by sampling these arrays at a fixed interval, and at each interval, the partial sums of the number of integers are computed. With these strategies, random cell access can be done in constant time. Note that, however, it may incur some overhead and this should be taken into consideration when used in a real-time application.

IV. EXPERIMENTAL RESULTS
In this section, we present the results for storing hyperspectral data with a k 2 -raster structure, incorporating the aforementioned improvements in padding storage and integer encoding strategies.
The hyperspectral scenes were captured by different sensors in real remote-sensing missions: Atmospheric Infrared Sounder (AIRS), AVIRIS, Compact Reconnaissance Imaging Spectrometer for Mars (CRISM), Hyperion, and IASI. These images are often used in the remote sensing data compression literature. All of them, save for IASI, are publicly available for download (http://cwe.ccsds.org/sls/docs/sls-dc/ 123.0-B-Info/TestData). The tested hyperspectral scenes are listed in Table I. Note that the storage size for hyperspectral scenes is measured in bits per pixel per band (bpppb).
We extended the algorithms presented in the paper by Ladra et al. [11], and our k 2 -raster implementation was based on these new extended algorithms. The DACs' software was downloaded from the Universidade da Coruña's Database Laboratory website (http://lbd.udc.es/research/DACS/). The original implementations for Simple-9, Simple-16, and PForDelta can be found on the website for the Poly-IR-Toolkit (https://code.google.com/archive/p/polyir-toolkit/source/default/source). However, to incorporate the random access function, the code was also modified and extended.
Programs for the experiments were written in C and C++ and compiled by GNU g++ 6.3.0 20170516 with -O3 optimization. The testing computer had an Intel(R) Xeon(R) 4-core CPU E3-1230 V2 @ 3.30 GHz with 8192 KB of cache and 32 GB of RAM. The operating system was Debian GNU/Linux 9 with kernel Linux 4.9.0-8-amd64. The software code is available at http://gici.uab.cat/GiciWebPage/downloads.php.

A. Storage Size With and Without Padding
In this section, we show how storage sizes fare among the different encoders using padded and unpadded matrices. The results are presented in Table I. It can be seen that k 2 -raster produces smaller storage sizes than gzip. The table also shows that the sizes for unpadded matrices are less than the padded matrices with up to 6% in savings. PForDelta using 128-integer blocks has the best results for the majority of padded and unpadded matrices, followed by DACs and Simple-16 encoders. Overall, the storage size has been reduced for most data except for Hyperion scenes using PForDelta, where PForDelta-128 codes for padded matrices produce similar results for PForDelta-256 codes for unpadded matrices. Both are almost equal, and the difference is not significant. If we examine Table I, it can be seen that the use of an unpadded matrix with PForDelta-128 compared with a padded matrix with DACs results in a reduction for storage for almost all the test files except for CRISM. We believe the experiments have shown us that using an unpadded matrix together with one of integer encoders, in particular PForDelta, can help improve the storage size.

B. Random Query With and Without Padding
In this section, we test the time it takes to query elements. Random access to elements in each tested scene is performed in 100 000 iterations with and without padding. The GetCell function in the paper by Ladra et al. [11] is modified using partial sums and sampling to optimize random cell access time in an unpadded matrix. To ensure more accurate results, we use the same set of coordinates and bands generated randomly for each scene for all the different encoders and matrices. The program is repeated 20 times for each scene and the average time is taken. The results are shown in Table II. It shows that the best access time is to use the 32-, 64-, and 128-integer PForDelta encoders for padded matrices and DACs, and 64-and 32-integer PForDelta encoders for unpadded matrices. What is more notable is that the access times are cut almost in half (49%) for most of the unpadded matrices, whereas AIRS Granules and IASI data have more or less the same access times for both types of matrices, possibly due to the similar tree data built from the padded and unpadded matrices. Other factors influencing execution times are the effectiveness of the partial sums and sampling optimization, and the relatively small spatial sizes of AIRS and IASI matrices (e.g., AIRS has 90×135 = 12 150 pixels) compared with others (e.g., AVIRIS Yellowstone has 680×512 = 348 160 pixels). Thus, the effects are less noticeable. Also, we should note that for access times in unpadded matrices, the way the tree is traversed is greatly boosted using the partial sums and sampling, which help access elements faster.
These experiments prove that the storage space and random access to elements in the k 2 -raster structure produce competitive results not only for DACs but also for the Simple family of word-aligned integer encoders.
V. CONCLUSION In this research, we propose a new no-padding method to reduce the storage space by saving only the elements in the nodes of a k 2 -raster that are within the bounds of the original matrix, and our experiments have shown that it saves space up to 6%. The access time has also been reduced by half for most of the data when using unpadded matrices. Furthermore, the use of other random access integer encoders, such as Simple-9, Simple-16, and PForDelta, has proven to be competitive compared with DACs, the encoder originally used by the authors of k 2 -raster. In particular, we can see that for most hyperspectral data, PForDelta performs better than DACs with up to 6% reduction in storage size and up to 20% reduction in random access time to elements. The experiments also show that the Simple family of integer encoders can also be used as a good alternative to DACs for random access to integer sequences.