EFINet: Efficient Feature Interaction Network for Real-Time RGB-D Semantic Segmentation

Real-time RGB-D semantic segmentation is crucial for tasks like dynamic environment analysis of mobile robots or autonomous vehicles, real-time and efficient segmentation results can enhance the accuracy of subsequent tasks such as free space detection, mapping and navigation. It requires models to balance computational cost and performance by employing more efficient mechanisms to effectively recognize differences in RGB-D multimodal information, retain complementary information, and reduce redundancies. Currently, although convolutional neural network (CNN) methods are less accurate than Transformer-based methods, they offer stronger real-time performance under the same computational load. Therefore, in this study, we proposed the Efficient Feature Interaction Network (EFINet), a real-time RGB-D segmentation method that uses a lightweight CNN encoder and incorporates encoder blocks with a lightweight upsampling method Dysample and the carefully optimized number of ConvNeXt V2 blocks, to redesign the decoder and minimize redundant computations. Additionally, we propose a robust cross-modal interaction mechanism that facilitates the exchange of useful information and the elimination of noise between modalities with minimal computational overhead. By incorporating lightweight convolutions, the proposed method can acquire multi-scale global information with minimal additional computation, thus providing the model with rich contextual information. The proposed EFINet has 38.0 M parameters and 40.0 GFlops, achieving 55.5% and 50.6% mIoU on the challenging NYUDv2 and SUN RGB-D datasets, respectively, and real-time inference at 31.2 FPS. EFINet outperforms several state-of-the-art methods in the real-time RGB-D segmentation task while achieving a better balance between segmentation performance and computational efficiency.