AMIANet: Asymmetric Multimodal Interactive Augmentation Network for Semantic Segmentation of Remote Sensing Imagery

In recent years, the inherent two-dimensional characteristics of optical images have led to a plateau in semantic segmentation performance. The complementary nature of Light Detection and Ranging(LiDAR) point clouds and camera images can effectively enhance semantic segmentation capabilities, thus, research into multi-modal joint semantic segmentation is garnering increasing attention. However, the domain gaps between different dimensions present challenges for the fusion of multi-modal data. In this paper, we introduce a novel Asymmetric Multi-modal Interaction Augmented Network (AMIANet), which directly processes heterogeneous data from images and point clouds. The treatment of the disparities in modal data ensures consistency in the features of both modes. Through the newly developed Synergistic Multimodal Interaction Module (SMI Module), AMIANet is capable of combining the complementary characteristics of cross-modal data. This is achieved by interactively fusing and extracting precise and rich structural information from point cloud features to enhance image characteristics. Experimental results on the N3C-California, WHU-RRDSD and ISPRS Vaihingen datasets demonstrate that AMIANet surpasses benchmark methods and current state-of-the-art approaches. The code will be available at https://github.com/2012153946/AMIANet.