Unpaired Dual-Modal Image Complementation Learning for Single-Modal Medical Image Segmentation

Objective: Multi-modal MR/CT image segmentation is an important task in disease diagnosis and treatment, but it is usually difficult to acquire aligned multi-modal images of a patient in clinical practice due to the high cost and specific allergic reactions to contrast agents. To address these issues, a task complementation framework is proposed to enable unpaired multi-modal image complementation learning in the training stage and single-modal image segmentation in the inference stage. Method: To fuse unpaired dual-modal images in the training stage and allow single-modal image segmentation in the inference stage, a synthesis-segmentation task complementation network is constructed to mutually facilitate cross-modal image synthesis and segmentation since the same content feature can be used to perform the image segmentation task and image synthesis task. To maintain the consistency of the target organ with varied shapes, a curvature consistency loss is proposed to align the segmentation predictions of the original image and the cross-modal synthesized image. To segment the small lesions or substructures, a regression-segmentation task complementation network is constructed to utilize the auxiliary feature of the target organ. Results: Comprehensive experiments have been performed with an in-house dataset and a publicly available dataset. The experimental results have demonstrated the superiority of our framework over state-of-the-art methods. Conclusion: The proposed method can fuse dual-modal CT/MR images in the training stage and only needs single-modal CT/MR images in the inference stage. Significance: The proposed method can be used in routine clinical occasions when only single-modal CT/MR image is available for a patient.