Spectral Reconstruction for Internet of Things Based on Parallel Fusion of CNN and Transformer

Spectral imaging acquires more analyzable and distinguishable information than RGB imaging and has become an emerging technique powering Internet of Things (IoT). Thus, generating spectral images from existing RGB cameras is instrumental to high level vision tasks. In this paper, an IoT-oriented spectral reconstruction model with parallel fusion of Convolutional Neural Network (CNN) and Transformer (PFCT) is proposed to efficiently recover hyperspectral images (HSI) from RGB counterparts, facilitating low-cost and non-hardware-specific spectral image acquisition. Recent works mainly utilize CNNs to extract local features by stacking more layers, ignoring the latent correlations of global features. In our PFCT network, we take advantage of lightweight CNNs to efficiently perceive local details, and exploit Transformer blocks to fully capture the global context. Based on the architecture, a parallel fusion module is further designed to deeply interact and fuse the features obtained by CNN and Transformer in both directions. The final output spectral image is generated with same spatial sizes of the original image based on the deeply fused features. Consequently, the proposed PFCT network achieved high performance on four benchmark datasets compared to several state-of-the-art networks with a relatively small number of parameters.