Prototype-Based Information Compensation Network for Multisource Remote Sensing Data Classification

Multisource remote sensing data joint classification aims to provide accuracy and reliability of land-cover classification by leveraging the complementary information from multiple data sources. Existing methods confront two challenges: interfrequency multisource feature coupling and inconsistency of complementary information exploration. To solve these issues, we present a prototype-based information compensation network (PICNet) for land-cover classification based on hyperspectral image (HSI) and synthetic aperture radar (SAR)/light detection and ranging (LiDAR) data. Specifically, we first design a frequency interaction module (FIM) to enhance the interfrequency coupling in multisource feature extraction. The multisource features are first decoupled into high- and low-frequency components. Then, these features are recoupled to achieve efficient interfrequency communication. Afterward, we design a prototype-based information compensation module (PICM) to model the global multisource complementary information. Two sets of learnable modality prototypes are introduced to represent the global modality information of multisource data. Subsequently, cross-modal feature integration and alignment are achieved through cross-attention computation between the modality-specific prototype vectors and the raw feature representations. Extensive experiments on three public datasets demonstrate the significant superiority of our PICNet over state-of-the-art methods. The codes are available at https://github.com/oucailab/PICNet