Transformer-Based Multi-Modal Deep Learning Framework for Early Disease Prognosis Using EHR and Medical Imaging Data

Early and accurate disease prognosis remains a critical challenge in modern healthcare, especially when clinical data is high-dimensional, heterogeneous, and incomplete. This paper proposes a novel Transformer-Based Multi-Modal Deep Learning Framework that synergistically integrates Electronic Health Records (EHR) and medical imaging data to predict disease progression at an early stage. Leveraging the power of Vision Transformers (ViT) for image modality and TabTransformer for structured EHR data, our model captures both spatial and contextual patterns, enabling robust prognostic predictions. The framework employs a cross-attention fusion module to align and integrate features from the two modalities, ensuring effective information transfer. We evaluate our model on two publicly available datasets: MIMIC-IV for EHR and NIH Chest X-ray14 for imaging. The results demonstrate significant improvements in AUC, F1-score, and accuracy compared to existing multi-modal and single-modality baselines. Additionally, we incorporate a SHAP-based interpretability module to provide transparent decision-making for clinicians. This study highlights the potential of transformer-based multi-modal learning to revolutionize early disease prognosis, supporting timely and personalized medical interventions.