Cross-Modal Visible-to-Infrared Image Translation in Remote Sensing Guided by Thermal Features

Weakly supervised visible (VI)-to-infrared (IR) image translation in remote sensing scenes is a challenging task that attempts to generate corresponding IR images from VI images with cross-model manner, thus tackling the acute scarcity of IR data in certain crucial task scenes. Existing methods frequently concentrate solely on style transfer, resulting in stylistic similarity but physical inconsistency, which limits the authenticity of cross-modal translation. IR thermal features stemming from brightness and darkness contrasts among distinct regions due to temperature disparities, are associated with visual representations in foreground objects in VI images. Therefore, this article proposes ThermalMask, a framework for IR image generation guided by thermal features, aiming to make the generated images more closely with IR features in remote sensing. Specifically, a saliency mask generation network (SMGN) and a semantic attention generation network (SAGN) are designed in the generator. The SMGN is used to generate background and foreground masks to preserve the background and most prominent features of the input remote sensing image. Meanwhile, the SAGN generates attention maps through pyramid nonlocal attention mechanism, guiding the network to adaptively focus on salient regions within both VI and IR feature maps. In addition, to refine the generated IR features, a pixel-spectrum multispatial constraint (PSC) targeting high-frequency components is designed, enabling the generative network to focus more on expressing IR features. Extensive experimental results have shown that our method has good performance in VI-to-IR image translation within remote sensing scenes, surpassing existing state-of-the-art image translation methods. Moreover, it holds promise for application in downstream tasks where IR data is insufficient.