DAE-GSP: Discriminative Auto-Encoder with Gaussian Selective Patch for Multi-modal Remote Sensing Image Classification

In the field of multi-modal remote sensing image (MRSI) classification, self-supervised learning algorithms have demonstrated significant advantages, particularly in scenarios with limited labeled samples. Existing self-supervised learning methods typically employ auxiliary tasks within either contrastive or generative frameworks, focusing on discriminative or structural information separately. In this paper, we propose a novel hybrid self-supervised learning paradigm, Discriminative Auto-Encoder with Gaussian Selective Patch (DAE-GSP) for MRSI classification. The DAE framework integrates contrastive learning with the Masked Image Modeling (MIM) technique, allowing for the simultaneous learning of structural information and discriminative representations from images. Furthermore, a cross-attention-based data-level fusion strategy is introduced during pretraining stage to enhance inter-modal interactions, thereby improving the effectiveness of modality fusion. Additionally, we propose a novel Gaussian Selective Patch (GSP) strategy, addressing the limitations of traditional square patch selection methods. Combined with self-supervised auxiliary tasks, this strategy facilitates the improved integration of multiple modalities and encourages the model to capture essential semantic information. Extensive experiments conducted on three public datasets (Houston2013, Augsburg, and Berlin) demonstrate the effectiveness of the proposed approach. With only 10 labeled training samples per class, the proposed method achieves overall accuracy of 90.15%, 82.64%, and 71.03% on the Houston2013, Augsburg, and Berlin datasets, respectively, indicating improvements of 1.31%, 1.22%, and 1.48% over state-of-the-art methods.