A Novel Multimodal Prototype Network for Interpretable Medical Image Classification

Medical image classification is a main task in medical diagnosis field. Some black box models have achieved expert-level accuracy on medical datasets, but these models are less adopted in clinical practice due to the lack of interpretability. In the past few years, designing interpretable models has been one of the major challenges in the medical field. The existing interpretable prototype networks only use medical images for training, ignoring the role of medical reports, and these medical reports can assist in prototype training and activation. Furthermore, existing prototype network methods neglect the position information in medical images, which is helpful for disease diagnosis. To solve these shortcomings, we propose an interpretable medical image classification framework (MProtoNet) that improves the accuracy and interpretability of disease predictions. In MProtoNet, we design a multimodal attention module and use prototype activation restriction loss to provide evidence for prototype training and activation. In addition, we design a position embedding module and multi-factor similarity calculation method to effectively utilize the position information in the image. We conducted experiments on the chest datasets MIMIC-CXR and open-I to test the model and compare it with other baseline models. Experimental results show that MProtoNet has made improvements in accuracy while preserving the interpretability of the model.