Sensorformer: A Memory-efficient Transformer for Industrial Sensor Fusion

The deep learning model, Transformer, and its variants have achieved state-of-the-art performance in many domains, such as natural language processing, vision recognition, protein structure prediction, as well as industrial multi-sensor data processing tasks. However, the high memory usage and $O(N^{2})$ computational complexity of Transformer can result in the high cost and low efficiency in industrial scenarios, especially in the case of edge computing. To mitigate this problem, we propose a novel auto-decorrelation block based on the Fast Fourier Transform and the self-attention mechanism of Transformer to reduce memory occupation and computational complexity. Instead of removing the related data, this model merges the correlated channels progressively throughout the whole inference process. The proposed method was tested on a public dataset based on condition monitoring of a hydraulic system. Results demonstrated that our proposed method reduced memory usage to around one-fifth of its original size while maintaining similar inference accuracy. The inference time can also be 2 times faster. This allows the transformer to be used for industrial multi-sensor data processing tasks in a more resource-efficient and faster manner.