Advancing Medical Image Diagnostics through Multi-Modal Fusion: Insights from MIMIC Chest X-Ray Dataset Analysis

In this study, we address the global scarcity of radiologists [19] by harnessing the synergy of image and textual data through deep learning. Utilizing the expansive MIMIC Chest X-ray dataset, we implement ResNet50 for image classification and BERT for sequence analysis. Our novel approach explores two multi-modal architectures: early and late fusion. The early fusion model pioneers the integration of image-text data at the input level, employing cross attention to deepen relational understanding. Contrastingly, late fusion synergizes extracted text and image features to augment diagnostic accuracy. Through rigorous experimentation, late fusion emerged superior, giving an impressive accuracy of 94.2% and an exact match ratio of 51.2%, outperforming both unimodality and early fusion benchmarks [6]. This study meticulously examines class imbalance in multi-label classification, innovatively applying different loss models, with dice loss leading in hyperparameter optimization via Optuna [1] across 50 trials. Despite the constraints of limited curated datasets, our models exhibit remarkable robustness. Our research not only advances AI tools for radiologists but also paves the way for accelerated patient care in medical imaging, marking a significant stride in multi-modal diagnostic methodologies.