Multimodal Fusion Model for Intelligent Diagnosis of Prostate Cancer Using Multiparametric MRI Clinical Report and Multiparametric US Images

Prostate cancer (PCa) is one of the most common malignant tumors in men worldwide. Multiparametric ultrasound (mpUS) and multiparametric magnetic resonance imaging (mpMRI) serve as common imaging modalities, providing clinically valuable information for PCa detection. Some evidence suggests complementary advantages of mpMRI and mpUS in lesion differentiation. This study developed a novel CLIP-based multimodal fusion architecture that effectively integrates features extracted from mpMRI clinical report and mpUS imaging data for PCa diagnosis. The proposed Intelligent diagnostic system achieved an accuracy of 0.847, precision of 0.865, recall of 0.898, F1-score of 0.881, and area under the receiver operating characteristic curve (AUC) of 0.912. The ablation experiments show that, on our dataset, the proposed method performs better on all evaluation metrics than the lack of any modality ultrasound images or the lack of valid mpMRI clinical report. This means that the information of mpUS and mpMRI used is effectively integrated by the model. In conclusion, the proposed method is a competitive intelligent diagnostic tool for PCa diagnosis.