Prototype-Driven and Multi-Expert Integrated Multi-Modal MR Brain Tumor Image Segmentation

For multi-modal magnetic resonance (MR) brain tumor segmentation, current methods usually directly extract the discriminative features from input images for tumor sub-region category. However, information aliasing caused by the mutual inclusion of tumor sub-regions is often ignored. Moreover, existing methods usually do not take tailored efforts to highlight the single tumor sub-region features. To this end, a multi-modal MR brain tumor segmentation method with tumor prototype-driven and multi-expert integration is proposed. It could highlight the features of each tumor sub-region under the guidance of tumor prototypes. Specifically, to obtain the prototypes with complete information, we propose a mutual transmission mechanism to transfer different modal features to each other. Furthermore, we devise a prototype-driven feature representation and fusion method, which implants the learned prototypes into tumor features and generates corresponding activation maps. With the activation maps, the sub-region features consistent with the prototype category can be highlighted. A key information enhancement and fusion strategy with multi-expert integration is designed to further improve the segmentation performance. The strategy can integrate the features from different layers of the extra feature extraction network and the features highlighted by the prototypes. Experimental results on three competition datasets prove the superiority of the proposed method. The source code can be available at https://github.com/Linzy0227/PDMINet.