Spatial Knowledge Transfer for Few-Shot Hyperspectral Image Classification Based on Residual Vision Transformers

Deep learning has significantly advanced the hyperspectral image (HSI) classification by leveraging extensive training data and model parameters. However, challenges persist due to limited and low-quality labeled samples from ground surveys, hampering performance. Additionally, training models with numerous parameters demands substantial computational resources. To overcome these challenges, we propose a spatial knowledge transfer method for few-shot HSI classification using a lightweight model. Our approach utilizes spatial prior knowledge from RGB images through transfer learning. We pretrain the spatial feature extraction network on mini-ImageNet datasets and fine-tune it on practical HSIs, resulting in significant improvements in classification accuracy and the model convergence speed. Moreover, we introduce an innovative network architecture that combines residual networks and Vision Transformers, effectively capturing both local features and long-range dependencies within patches to extract global-local spatial information. Complementing spatial features, we employ a spectral feature extraction network based on one-dimensional convolutions, chosen for its effectiveness in capturing spectral characteristics. The extracted spatial and spectral features are seamlessly integrated using a fusion block based on fully connected networks. Experimental results on two publicly benchmarked HSI datasets demonstrate the superior performance of our method over existing few-shot and supervised learning approaches.