SKD-Net: Spectral-based Knowledge Distillation in Low-Light Thermal Imagery for robotic perception

Enhancing the generalization capacity for semantic segmentation of aerial perception systems for safety-critical applications is vital, especially for environments with low-light and adverse conditions. Multi-spectral fusion techniques aim to maintain the merits of electro-optical (EO) and infrared (IR) images, e.g., retaining low-level features and capturing detailed textures from both modalities. However, these techniques encounter limitations when faced with scenarios involving missing modalities, especially during inference when only IR images are available. In this paper, we propose a novel spectral-based knowledge distillation architecture known as SKD-Net to improve the performance of deep learning models for missing modality scenarios for semantic segmentation tasks. In this architecture, we make use of Gated Spectral Unit to combine information from both modalities. SKD-Net aims to extract valuable semantic information from EO images while preserving spectral knowledge from the IR images within the feature space. The model retains the style information in the shallow layers while simultaneously fusing the high-level semantic context obtained from EO and IR images to improve the feature generation capacity when dealing with only IR images during inference. SKD-Net outperforms state-of-the-art multi-modal fusion and distillation models by 2.8% on average in scenarios with missing modalities when using only IR data during inference in two public benchmarking datasets. This performance increase is achieved without additional computational costs compared to the baseline segmentation models.