Aggregable generalized deduplication

Considering the edge-cloud environment and uploading large amounts of data from IoT devices through the edge node towards the cloud, this paper investigates an aggregation method of data streams compressed by Generalized Deduplication (GD) for the edge node. The simplest way is to decode GD-compressed data streams, aggregate raw streams and re-encode it into a single GD-compressed data stream. However, this involves a large computational complexity for aggregation due to the decoding and re-encoding of linear codes underlain the GD. From the above observation, this paper presents a novel aggregation method, called Aggregable GD (AGD) to aggregate multiple GD streams into a single compressed stream. AGD consists of two main key ideas: The first one is to reduce the computational complexity of elimination of the redundancy among multiple GD streams by partly extracting data from the compressed stream. The second one is to create an additional dictionary corresponding to each leaf node to eliminate redundancy within a single stream from a single leaf node. By applying these, in addition to the reduction of computational complexity, the compression performance is also expected to be improved compared to the ordinary decoding-aggregating-re-encoding method. For this expectation, this paper clarifies that AGD dramatically improves the computational complexity of aggregation by mathematical estimation. Furthermore, by the computer simulation using the Reed-Solomon code as the underlying linear code of GD, we demonstrate that AGD performs better than the ordinary method from the viewpoint of the compression rate.