Fine Tuning of Large Models for Hyperspectral and LiDAR Data Joint Classification

Pre-trained visual language models (VLMs) excel in various visual tasks due to the extensive knowledge they gain from large datasets of image-text pairs. The ability of VLMs to scale effectively is a major benefit, enabling their application across a diverse array of downstream tasks. This study focuses on leveraging VLMs to jointly classify hyperspectral image (HSI) and LiDAR data, facilitating the extraction of more generalized features from various remote sensing image sources. Firstly, this study employs a feature extractor to convert the low-dimensional patches of HSI and LiDAR data into high-dimensional latent feature representations, aligning with the VLMs' requirements for the dimensionality of visual input data. VLMs use continuous vectors for classification, generated by text template prompts with class names. However, prompt engineering often demands considerable specialized knowledge and time. To address this challenge, this paper introduces Prompt Tuning (PT) and Residual Tuning (RT) methods. PT enhances model performance by incorporating random prompts into the visual encoder, while RT introduces randomly initialized learnable text residuals (TRs) into text features, further improving the generality of VLMs. The experimental results indicate that the proposed method attains competitive classification accuracy on the Houston 2013 dataset, even with a limited quantity of labeled pixels.