MSDT: Multiscale Diffusion Transformer for Multimodality Image Fusion

Multimodal image fusion is a vital technique that integrates images from various sensors to create a comprehensive and coherent representation, with broad applications in surveillance, medical imaging, and autonomous driving. However, current fusion methods struggle with inadequate feature representation, limited global context understanding due to the small receptive fields of convolutional neural networks (CNNs), and the loss of high-frequency information, all of which lead to suboptimal fusion quality. To address these challenges, we propose the Multi-Scale Diffusion Transformer (MSDT), a novel fusion framework that seamlessly combines a latent diffusion model with a transformer-based architecture. MSDT uses a perceptual compression network to encode source images into a low-dimensional latent space, reducing computational complexity while preserving essential features. It also incorporates a multiscale feature fusion mechanism, enhancing both detail and structural understanding. Additionally, MSDT features a self-attention module to extract unique high-frequency features and a cross-attention module to identify common low-frequency features across modalities, improving contextual understanding. Extensive experiments on three datasets show that MSDT significantly outperforms state-of-the-art methods across twelve evaluation metrics, achieving an SSIM score of 0.98. Moreover, MSDT demonstrates superior robustness and generalizability, highlighting the potential of integrating diffusion models with transformer architectures for multimodal image fusion.