Differential Modal Multistage Adaptive Fusion Networks via Knowledge Distillation for RGB-D Mirror Segmentation

Mirrors play a significant role in our daily lives and are ubiquitous. However, deep learning computer vision models find them challenging owing to the negative impact of reflected information on scene understanding. This study addresses two key challenges faced by multimodal models. First, the cross-modal variability of features at different stages is generally overlooked by contemporary backbone networks. Second, good performance has only been achieved at an unacceptable computational expense, owing to the numerous parameters used. To address the first challenge, we propose a differential-mode multistage adaptive fusion network (differential mode refers to images generated by different sensors that are differentiated to complement each other) that incorporates two-step fusion in the coding stage to account for the degrees of difference among the cross-modal features. In the first stage, wherein considerable differences in modal features exist, multi-angle fusion is performed. In the second stage, wherein the differences are smaller, a hierarchical adaptive fusion strategy is employed. Regarding the second challenge, we introduce a companion training framework for mirror segmentation that combines knowledge distillation and contrastive learning. Our proposed scheme achieves state-of-the-art performance on an available mirror segmentation dataset without requiring numerous parameters.