Triplet Contrastive Learning Framework With Adversarial Hard-Negative Sample Generation for Multimodal Remote Sensing Images

Supervised learning models have achieved remarkable success in the field of remote sensing, but their applicability is limited by the significant requirement of high-quality labeled data. This article presents the triplet adversarial contrastive learning (TACL) model, as a self-supervised feature extractor. Considering the potential contrastive semantic conflicts, which may occur due to the different descriptive abilities of various modalities, TACL constructs a triplet contrastive learning framework aligning the two original modalities and a fused modality. To augment the acquired representations of the model, TACL introduces an adversarial hard-negative sample generation (AHSG) strategy, aiming to boost the resemblance between the feature vectors of negative samples and anchors. In addition, a ConvNeXt-based lightweight encoder is designed as the foundational backbone of the model, specifically enriching the expression of central features. A series of few-shot classification experiments substantiate the exceptional performance of the features extracted by TACL, with the simplistic classifier support vector machine (SVM). As a label-free pretraining approach, TACL holds great potential for enhancing the performance of various multimodal remote sensing tasks in scenarios with limited label availability.