DECT: Diffusion-Enhanced CNN–Transformer for Multisource Remote Sensing Data Classification

Methods for joint classification of hyperspectral images (HSIs) with high dimensionality and spectral correlation and other sensor data (e.g., optical, infrared, radar, etc.) are important directions in the field of remote sensing. To better learn the feature representation of diffusion features (HSI), the unsupervised global modeling property of diffusion is utilized to mine the potential features of HSI to obtain diffusion features as input data. In addition, to fuse HSI features, HSI diffusion features, and other data features, a three-input diffusion-enhanced CNN–transformer (DECT) network based on CNN and transformer is proposed for feature extraction and fusion. First, the primary features are extracted by hierarchical CNN after premodal fusion. Second, considering the high dimensionality of HSI, spectral pooling attention interaction is designed for feature extraction and aggregation of information from different attentions. Finally, the inverted bottleneck convolutional transformer is designed to aggregate multisource information to enhance feature reuse and aggregate local and contextual information. It is shown on three publicly available datasets that DECT outperforms current state-of-the-art methods.