Spectral-Spatial Dual-Branch Selective Fusion Enhanced Transformer for Hyperspectral Image Classification

In recent years, deep learning approaches that integrate convolutional neural networks (CNNs) with Transformer architectures have greatly enhanced the accuracy of hyperspectral image (HSI) classification. However, the combination of these two methods has not fully explored the deep correlations between spectral and spatial features nor effectively utilized the discriminative information from the central pixels. To address these issues, we propose a novel spectral-spatial dual-branch selection fusion enhanced Transformer (DSFEFormer) network. Specifically, a spectral-spatial dual-branch structure is designed to extract comprehensive and rich spectral and spatial features, thereby enhancing the feature representation capability of the image. Subsequently, a spectral spatial selection attention (SSSA) is introduced to guide the deep fusion of spectral and spatial features, adaptively focusing on the most discriminative regions while effectively suppressing redundant information. Additionally, to fully leverage the deep fusion features, we design a Center Manhattan Transformer Encoder (CMTE) composed of two key modules, Pooling Manhattan Attention (PMA) and convolutional gated feed forward network (CGFFN) to model the relationships between central pixels and their neighboring pixels effectively, further enhancing the ability to perceive subtle feature differences. Comparative experiments and analyses are conducted on four publicly available datasets—Pavia University, Salinas, Xuzhou, and WHU-Hi-LongKou. The experimental results indicate that DSFEFormer achieves superior classification performance while maintaining low computational complexity, and also exhibits certain robustness and generalization potential.