Semantic Attention Mechanism-based Infrared and Visible Image Fusion Method

Infrared and visible image fusion aims to integrate the images captured by different types of sensors to generate fused images with abundant texture details and salient target information. Recently, there is a growing emphasis that the fused images can facilitate the downstream high-level visual tasks. However, current semantic-driven fusion algorithms either do not fully exploit the potential of semantic labels or lack interpretability. To overcome these limitations, we present a semantic attention mechanism-based infrared and visible image fusion method, termed as SAMFusion. Specifically, a fusion network and a semantic segmentation network are constructed. In the fusion network, the Wavelet Residual Gradient Convolution module (WRGC) is developed to extract global features while enhancing the fine-grained information. Furthermore, the Self-to-Cross Attention Fusion module (SCAF) is designed in order to integrate complementary information and reduce redundancy. For the network training, we develop a weighted intensity loss based on semantic attention mechanism according to semantic labels. It can selectively constrain pixels of different classes to align with a generating reference value. Extensive experiments on widely used datasets show that our SAMFusion achieves better performance than state-of-the-art semantic-driven fusion methods in both qualitative evaluation and quantitative measurement.