Multilevel Multimodal CNN–Transformer Fusion Network for Land Use and Land Cover Classification

In remote sensing (RS), convolutional neural networks (CNNs) are well-recognized for their spatial–spectral feature extraction capabilities, whereas vision transformers (ViTs), which leverage attention mechanisms in mining long-range dependencies, have become prominent in land use and land cover (LULC) classification (CLS). As single-modal LULC approaches easily encounter performance bottlenecks, multimodal information fusion methods have gained increasing attention because of their potential to overcome these limitations. However, existing multimodal fusion methods are typically restricted to a single level, that is, they either extract multimodal features from single images or derive single features from multimodal images without fully leveraging the multimodal characteristics at both the image and feature levels. To this end, we propose a novel multilevel multimodal CNN–transformer (MCT) fusion network that simultaneously integrates multimodal information at both the image and feature levels. The MCT network incorporates a multifeature extractor (MFE) to utilize the powerful spatial–spectral information extraction of the CNNs alongside the global representation strengths of transformers. Additionally, a cross-architecture fusion (CAF) module is designed to facilitate a comprehensive fusion of multimodal information across architectures and levels. We conduct extensive experiments on three multimodal benchmark datasets, i.e., the Houston2013 dataset containing hyperspectral (HS) and light detection and ranging (LiDAR) data, the Berlin dataset comprising HS and synthetic aperture radar (SAR) data, and the Augsburg dataset consisting of HS, SAR, and digital surface model (DSM) data, to demonstrate that our proposed method consistently outperforms state-of-the-art (SOTA) methods and to underscore the importance of multimodal information fusion for the advancement of LULC CLS.