Semantic Region Adaptive Fusion of Infrared and Visible Images via Dual-DeepLab Guidance

Given the potential diminishment of semantic information caused by previous nonregion-specific maximum or weighted intensity losses under different lighting conditions, we propose a novel high-level semantic perception fusion framework, termed semantic region adaptive fusion (SRAFusion). We establish a prioritized preservation scheme for high-level semantic information (HLSI), gradient information, and intensity information, ranked in descending order of priority. Based on this prioritization scheme and prior knowledge of semantic distribution in the source images, we construct a ground truth for the fusion task. Specifically, we capture the HLSI distribution of the source images using two independent semantic segmentation networks. Subsequently, we introduce semantic region decision block (SRDB) to partition the original scene into region with bimodal HLSI, region with unimodal HLSI, and region lacking HLSI. We then design specific loss functions to constrain the aforementioned regions, facilitating the integration of complete semantic information. Furthermore, taking into account the susceptibility of the visible segmentation network to lighting conditions, we use a two-stage training strategy involving coarse-tuning and fine-tuning. This method aims to optimize one-stage training strategy and achieve a more accurate region delineation. Finally, qualitative and quantitative experiments conducted on publicly available datasets such as MFNet, RoadScene, and TNO demonstrate the superiority of our SRAFusion over state-of-the-art methods. Our code will be available: https://github.com/WenziCao/SRAFusion.