Huffman Deep Compression of Edge Node Data for Reducing IoT Network Traffic

Data compression at the Internet of Things (IoT) edge node aims to minimize data traffic in smart cities. The traditional Huffman Coding Algorithm (HCA) is shown as the most effective compression algorithm for sensor data. However, implementing the algorithm at IoT edge nodes is hindered due to memory limitations; HCA requires a large amount of memory to construct a Huffman tree to compress data. To address this issue, this paper proposes a new lossless Huffman Deep Compression (HDC) algorithm that incorporates the sliding window technique to fit in memory, reduces the complexity of the Huffman tree using deep learning pruning and pooling techniques, and uses pattern matching with pattern weights instead of using symbol matching and symbol frequencies in HCA. This paper introduces a sliding window approach to minimize memory usage, leveraging pattern matching and weights for higher compression and employing deep learning techniques to reduce the Huffman tree size through pruning and pooling. Experiments were performed using the Esp8266 MCU IoT node on eight numerical attributes from sensors of six of Malaysia’s air pollution station datasets. The findings demonstrate that the HDC algorithm has substantially reduced data size (p-value<0.0005), achieving a higher compression ratio (CR) by 1.4x while reducing data size by up to 59%. Furthermore, this achievement is attained while utilizing less than 80 KB of IoT memory and consuming at most 44 mAmps per slide compression. Moreover, the compression performance correlated linearly with the number of patterns in each sliding window. With such excellent performance, using HDC at IoT edge is a considerable solution to reduce the smart-cities network traffic.