CACFTNet: A Hybrid Cov-Attention and Cross-Layer Fusion Transformer Network for Hyperspectral Image Classification

Hyperspectral (HS) image classification has become an important research area. Although previous work on HS image classification has achieved impressive results, finding a proper balance between extracting spatial–spectral information and capturing band similarities remains a challenge. To address this problem, we designed some efficient modules and constructed hybrid covariance attention and cross-layer fusion transformer network (CACFTNet), which efficiently models the extraction of spatial–spectral information and band similarity. First, our approach combines local and global perspectives to process the desired feature information. We introduce the dual-branch feature processing (DBFP) module, which can model the desired spatial–spectral information and band similarity. Second, we design the dual-branch feature fusion (DBFF) module, which combines a convolutional neural network (CNN) and a transformer to catch spatial–spectral information at different scales and fuse them effectively. To further process the features, we construct the unparameterized covariance attention (UPCA) module, which utilizes the covariance matrix to capture the correlation between different spectral bands. This allows the network to concentrate on bands that are more useful for classification tasks. In addition, we designed a hybrid activation function (HAF) that maps the channel values to specific ranges and emphasizes the extent to which the correlation varies between different bands. Finally, in order to incorporate important information from different layers, we propose the cross-layer adaptive attention fusion (CAAF) module, which fully fuses information between layers and enriches the overall information representation. We evaluate our proposed model on three well-known public datasets and demonstrate its superiority over existing approaches.