Ternary Modality Contrastive Learning for Hyperspectral and LiDAR Data Classification

In the domain of remote sensing image classification, single sensors are constrained by their sensing angles and information dimensions, rendering them incapable of fully capturing the intricate characteristics of ground objects. Different sensors can provide complementary information, significantly enhancing the performance of object classification. However, due to their unique physical observation principles and varying spatial-spectral resolutions, different modalities capture heterogeneous features of ground objects, resulting in a semantic gap issue when integrating modalities. This article designs a multimodal contrastive learning framework, starting with the preprocessing of hyperspectral image (HSI) to obtain its spatial and spectral modalities, and then combining these with the light detection and ranging (LiDAR) modality, forming a ternary modality contrastive learning framework that achieves deep semantic alignment between different modalities. Furthermore, to enhance the model’s generalization ability, based on the neighborhood semantic similarity of HSI, we propose a spectral selection data augmentation method. Extensive experiments on four public datasets show that our method outperforms several other state-of-the-art (SOTA) methods in classification performance.