H2MWSTNet: A Hybrid Hierarchical Multigranularity Window Shift Transformer Network for Hyperspectral Image Classification

Currently, hybrid network models combining convolutional neural networks (CNNs) and Transformers have garnered significant interest among researchers in hyperspectral image (HSI) classification. However, some hybrid models only consider the combination of a single-branch CNN and Transformer, neglecting the shallow multiscale spatial-spectral information and the deeper local spatial information. To address these issues, we propose a hybrid hierarchical multigranularity window shift transformer network (H2MWSTNet), which can efficiently extract multiscale spatial-spectral information and capture local spatial information in sequences. Specifically, we construct a new dual-branch spatial spectral convolution (DBSSC) block. The block effectively realizes the fusion of spatial-spectral information at different scales by aggregating the outputs of both spatial and spectral branches. Furthermore, we have designed a global feature enhancement (GFE) module that effectively enhances the features of the input token sequences. Along with GFE, we propose a multigranularity window shift attention (MWSA) module, which dynamically shifts the key matrix, enhancing the distinguishability of deeply extracted local spatial features. The GFE and MWSA modules provide a novel perspective for HSI feature extraction, constituting a feature mixer based on GFE and multigranularity window shift attention (GMSA) module. Building on the generic architecture and GMSA, we further developed a multigranularity window shift transformer (MWSFormer) encoder block. Finally, our CNN-Transformer hybrid model is designed in a hierarchical fashion, effectively reducing dimensionality and markedly improving classification accuracy. We conducted extensive experiments on six public HSI datasets, and the results demonstrate that H2MWSTNet outperforms the most advanced classification algorithms.