CSPA-GAN: A Cross-Scale Pyramid Attention GAN for Infrared and Visible Image Fusion

Infrared and visible image fusion (IVIF) aims to combine high contrast of infrared image and rich texture details of visible image, which can break through the imaging limitations of single modality. Generative adversarial network (GAN) has recently received lots of attentions on IVIF due to the adversarial learning without requiring paired multimodality image and label image. Nevertheless, the existing GAN-based approaches are still severely affected by information bias between infrared image and visible image, and it may result in unnatural visual effects. To mitigate the issue, this article proposes a novel cross-scale pyramid attention GAN-based IVIF method (CSPA-GAN), which adopts one generator and dual discriminators to approximate the distribution of fused image. The generator is composed of head module, pyramid decomposition path, feature fusion path, decoding path, and reconstruction module. First, the low-level features, extracted by head module, are further decomposed into multiscale features though pyramid decomposition path. In the feature fusion path, we develop a residual attention weight fusion rule (Res-AWFR) to fuse the multiscale features at each scale. The decoding path with bidirectional interactions decodes the fused features pyramid, which is constructed by long short-term memory module and cross-scale pyramid attention (CSPA). Finally, the reconstruction module produces the fused image. Comparing with current popular deep learning (DL)-based methods, our CSPA-GAN delivers high-performance gains on the TNO, INO, and MSRS datasets qualitatively and quantitatively.