Robust One-Stop Multi-Modality Image Registration-Fusion-Segmentation Framework Against Misalignments and Adversarial Attacks

In complex open scenes, multi-modality image fusion and segmentation encounter two challenges: i) Imaging misalignments, manifested as pixel shifts and structural distortions, are perceptible. ii) Human-crafted adversarial attacks, reflected in pixel distribution variations, are imperceptible. They not only degrade the visual quality of fused images, e.g., noticeable edge ghosts but more critically undermine semantic perception. However, none of the existing works considered the coupled effect of these degradations. This paper proposes a One-Stop framework incorporating sequential task flows of “Registration-Fusion-Segmentation”, termed OS-RFS. Registration aims to mitigate the chained impact of misalignment on fusion and segmentation. We follow a coarse-to-fine registration paradigm and develop a Global-Local Incremental Registration (GLoIR) model, where the global shift registration (GSR) is performed initially for long-range pixel shifts, followed by incremental local deformation registration (LDR) for subtle local deformations. To improve segmentation robustness, we innovatively introduce auxiliary positive attacks and build a Cancellation Defense Strategy (CDS) in the fusion model. The CDS constrains the fusion model to fit fused images to the distribution of positive attacks, endowing fused images with a robust defense ability against adversarial attacks. This significantly mitigates the impact of adversarial attacks on semantic segmentation. Extensive experimental results reveal that our OS-RFS performs remarkable robustness on multi-modality image fusion and semantic segmentation against imaging misalignments and adversarial attacks.