Dual-Prototype Learning With Multisemantic Fusion for Cross-Domain Few-Shot Hyperspectral Image Classification

Multimodal learning has demonstrated outstanding performance in various fields, including visual tasks, due to its ability to integrate information from different data sources. In the field of hyperspectral image (HSI) classification, although existing research can handle complex image data, the utilization of semantic information is insufficient. Over-reliance on a single image modality overlooks the synergistic effects of cross-modal information, thereby limiting the model’s performance in category center representation and discrimination, especially in scenarios with scarce samples. To address these challenges, this article proposes a dual-prototype learning with multisemantic fusion (DPL-MSF) method. Specifically, within the DPL-MSF framework, precise text prototypes are constructed by leveraging category textual information and prompt learning strategies to capture the first layer of semantic information. In addition, soft labels (SLs) are introduced into the image prototypes, and a feature fusion module is designed to deeply integrate SL information, thereby obtaining another layer of semantic information. The primary role of the text prototype is to assist in the generation of image prototypes before the introduction of SLs, through contrastive learning methods. Under the synergistic effect of multiple semantic information, the final class-level prototypes generated are more representative and discriminative, significantly enhancing the performance of few-shot learning. Experiments conducted on multiple standard HSI datasets have shown that DPL-MSF has significant advantages in scenarios with scarce samples and cross-domain settings. The code will be available at https://github.com/AIYAU/DPL-MSF