Spectral Query Spatial: Revisiting the Role of Center Pixel in Transformer for Hyperspectral Image Classification

Recently, there have been significant advancements in hyperspectral image (HSI) classification methods employing Transformer architectures. However, these methods, while extracting spectral-spatial features, may introduce irrelevant spatial information that interferes with HSI classification. To address this issue, this article proposes a spectral query spatial transformer (SQSFormer) framework. The proposed framework utilizes the center pixel (i.e., pixel to be classified) to adaptively query relevant spatial information from neighboring pixels, thereby preserving spectral features while reducing the introduction of irrelevant spatial information. Specifically, this article introduces a rotation-invariant position embedding (RIPE) module to integrate random central rotation (RR) and center relative position embedding (PE), mitigating the interference of absolute position and orientation information on spatial feature extraction. Moreover, a spectral-spatial center attention (SSCA) module is designed to enable the network to focus on the center pixel by adaptively extracting spatial features from neighboring pixels at multiple scales. The pivotal characteristic of the proposed framework achieves adaptive spectral-spatial information fusion using the Spectral Query Spatial paradigm, reducing the introduction of irrelevant information and effectively improving classification performance. Experimental results on multiple public datasets demonstrate that our framework outperforms previous state-of-the-art methods. For the sake of reproducibility, the source code of SQSFormer will be publicly available at https://github.com/chenning0115/SQSFormer.