Mixing Self-Attention and Convolution: A Unified Framework for Multisource Remote Sensing Data Classification

Convolution and self-attention are two powerful techniques for multisource remote sensing (RS) data fusion that have been widely adopted in Earth observation tasks. However, convolutional neural networks (CNNs) are inadequate for fully mining contextual information and representing the sequence attributes of spectral signatures. In addition, the specific self-attention mechanism often comes with high-computational costs, which hinders its application in the field of RS. To overcome the above limitations, this article proposes a unified framework called “mixing self-attention and convolution network (MACN)” for comprehensive feature extraction and efficient feature fusion. First, the proposed MACN utilizes two adaptive CNN encoders (ACEs) to extract shallow convolutional features from multisource RS data. Second, taking the complexity and varying scales of RS data into account, the proposed mixing self-attention and convolution Transformer (MACT) layer achieves local and global multiscale perception through an elegant integration of self-attention and convolution. MACT can extract abundant spatial and high-dimensional information (e.g., spectral and elevation information) while maintaining minimal computational overhead compared with pure convolution or self-attention counterparts. Finally, a multisource cross-guided fusion (MCGF) module is designed to achieve deep fusion of multisource RS data features. MCGF utilizes a carefully designed cross-modal attention mechanism to capture the interaction between multisource data and aggregate contextual information. Extensive tests on six public RS datasets have shown that our method outperforms other multisource fusion models, delivering state-of-the-art (SOTA) results on multiple RS data fusion tasks without specific tuning. The source code of the proposed method will be available publicly at
https://github.com/like413/MACN
.