Multi-Resolution Infrared-Visible Image Fusion using Multi-Scale Residual Quantization

Infrared-visible image fusion (IVF) is an essential task in multimodal image processing that integrates infrared and visible modalities to enhance the overall image information content. However, existing methods often suffer from limited precision and efficiency. Furthermore, they fail to address practical requirements such as multi-resolution fusion and mutual translation. In this paper, we propose the Multi-Scale Residual Quantized Infrared-Visible Image Fusion (M-RQIVF) framework to efficiently generate high-quality fusion images. M-RQIVF trains multi-scale residual quantized infrared and visible autoencoders that convert images into multi-scale discrete token maps. This approach approximates the residuals from the features on a scale-by-scale basis, allowing for coarse-to-fine fused image generation that aligns well with human visual perception. Furthermore, by leveraging these discrete token maps, we train Visual Auto-Regressive (VAR) transformers using next-scale prediction. The VAR transformer ensures that features of corresponding sizes can be generated, even when the input infrared and visible images have different resolutions, facilitating fine-grained fusion. Additionally, the autoregressive structure enables image translation to be treated as a conditional generation task, thereby enabling mutual translation between infrared and visible images. Extensive experiments demonstrate that M-RQIVF outperforms the SOTAs while maintaining a much faster inference speed.