Multiattentive Perception and Multilayer Transfer Network Using Knowledge Distillation for RGB-D Indoor Scene Parsing

Scene parsing has gained wide attention in the field of computer vision, with emerging methods and techniques providing superior solutions. Although some methods have improved performance, they tend to neglect the number of model parameters and computational size, which makes achieving real-time operation in practical applications challenging. To address these limitations, we propose a multiattentive perception and multilayer transfer network that employs knowledge distillation (MPMTNet-KD), which is generated by a student network (MPMTNet-S) under the guidance of a teacher network (MPMTNet-T) with the aid of our proposed multilayer transfer knowledge distillation (KD) methods. To capture complete information from different modalities, a multiattentive perception module (MAPM) is introduced to mine features from various perspectives, and hetero-oriented sensing (HOS) convolution is utilized to integrate cross-layer features in a single and holistic manner. Importantly, we introduce multilayer transfer KD to explore the different knowledge types between layers, as well as intraclass and interclass correlations. In addition, we use the discrete cosine transform (DCT) approach combined with filtering during the KD process to mitigate noise that may be induced by the depth map, thereby improving the depth information and further enhancing the knowledge transfer effect. We conducted comprehensive experiments on two challenging indoor benchmark datasets, namely NYUDv2 and SUN RGB-D. Compared with existing methods, the proposed MPMTNet-KD reduces the number of parameters from 125.8 M in MPMTNet-T to 28.3 M in MPMTNet-S, achieving a mean intersection over union (mIoU) of 54.9% in the indoor scene parsing task. MPMTNet-KD was also evaluated on two additional public datasets, namely MFNet and PST900, to demonstrate its generalization capacity. The source code is available at https://github.com/XUEXIKUAIL/MPMTNet