Hyperspectral Image Classification Based on a Locally Enhanced Transformer Network

Recently, transformer-based models have achieved remarkable performance in hyperspectral image (HSI) classification. However, due to the limited training data, existing methods often show limited capability of capturing fine-grained local features. Although attempts have been made to solve this problem, the large amount of parameters imposes the risk of overfitting. In this article, we propose a locally enhanced transformer network for HSI classification with fewer network parameters, which mainly consists of a multibranch spatial-spectral tokenization (MSST) module and a dual-branch transformer encoder (DTE) module. The MSST generates effective spatial-spectral tokens through diverse convolutions with a residual connection. The DTE consists of a global transformer branch and a locally enhanced transformer branch, which are used to capture the global and local spatial dependencies of HSI, respectively. Unlike the conventional self-attention (SA) module used in the global branch, we propose an improved multihead SA (IMSA) module in the local branch by incorporating the local prior information of HSI with graph convolution, to enhance the local information extraction. To fuse the global and local features from the two branches, we introduce an adaptive strategy by using learnable weights for both branches. We devise our MSST and DTE with a shallow architecture, significantly reducing the number of parameters. Experimental results on benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art.