An Entropy-Bounded, General, Model-Based Framework for Lossy Compression of Sensor Data

In many industries, digital twinning has become an indispensable element of advanced technologies. However, digital twins are heavily reliant on extensive Internet of Things (IoT) sensor measurement data to function effectively. Consequently, data mining has become a lucrative endeavor, akin to gold rushes in the XIX century. However, the substantial volume of collected data often stresses the storage capacities for smaller to medium-sized enterprises, necessitating efficient compression techniques. Error-bound lossy compression offers substantial data reduction advantages, but introduces distortion that, when uncontrolled, can adversely affect analysis. This paper proposes an information optimization scheme that employs information entropy as a comprehensive measure of distortion across lossily reconstructed sensor data. Moreover, we present a general model-based compression framework capable of extracting the uniquely observed data features prior to compression. The results of our analysis show that the proposed framework further improves compression performance over direct compression of raw data.