Multimodality-Assisted Semi-Supervised Brain Tumor Segmentation in Nondominant Modality Based on Consistency Learning

Multimodal magnetic resonance imaging (MRI) provides a wealth of complementary information for determining the anatomical structure and pathological features of tumors. However, due to the limitations of patient progression and imaging cost, complete acquisition of MRI in various modalities is often not possible in a clinical setting. To achieve segmentation effect in nondominant modalities that is achieved in dominant modalities, encouraged by potential value of intrinsic connections between different modalities and considering the difficulty of pixel-level annotation of medical images, we propose a win–win approach called multimodality-assisted semi-supervised segmentation network (M2S3-Net). The core of our proposed approach is a multinondominant modality-assisted semi-supervised training strategy, which extracts generic and robust modality features from a limited annotated image by learning implicit features between two nondominant modalities to achieve information complementarity. To accommodate the above framework, we further propose modality fusion module (MFM) and cross-modality-assisted (CMA)-skip connection, which adaptively aggregate modality-independent features in a learnable manner to enhance the representativeness of deep models. Experiments on the public dataset BraTS2019 show that for the segmentation of perineural edema, using two nondominant modalities, the proposed method achieves up to 77.82% (10% labeling) and 78.33% (20% labeling) of the dice coefficients (DICEs), which is over 10% improvement compared to segmentation networks based on semi-supervised learning (SSL) that utilizes only a single nondominant modality. To a single dominant modality, it is 6.14% and 6.35%, respectively. Compared with other multimodal segmentation methods, our method achieves 83.55% in tumor core (TC) using only 20% labels, which is superior to previous methods.