Clinical Diagnosis Using Multi-Modal Transformer

Improving the accuracy of medical diagnosis requires integrating multi-modal data comprised of medical images and clinical records. Traditional deep learning models often fail to combine these sources effectively. This study proposes a multi-modal transformer-based framework integrating textual and visual medical data to enhance diagnostic accuracy. The proposed model employs a dual-encoder transformer with cross-modal attention. The system processes medical images through Vision Transformer (ViTs), converts them into image tokens and clinical data, through BERT converts data into text tokens separately, and are fused through attention mechanisms. When tested on lung disease data, it reached a BLEU score of 0.45 for report generation, showing it can produce high-quality, readable medical reports, which help clinicians make the medical diagnosis.