Transforming Heart Disease Detection with BERT: Novel Architectures and Fine-Tuning Techniques

The prediction of heart disease is crucial for effective prevention and treatment. However, extracting clinical infor-mation such as CAD, smoking, hypertension, hyperlipidemia, obesity, and family history of CAD from unstructured electronic health records (EHRs) poses significant challenges to clinicians. This research introduces a novel approach that leverages an ensemble of transfer learning algorithms combined with a multi-head attention mechanism to automatically extract heart disease risk factors from EHRs. Various deep learning models, including BERT, BioBERT, BioClinical BERT, RoBERTa, and XLNet, were initially trained on medical data sets and subsequently fine-tuned on the i2b2 clinical data set. Individual models delivered strong results, with RoBERTa achieving the highest accuracy of 95. 27% and an F1 score of 94. 94%. BioBERT, BioClinical BERT, XLNet, and BERT also performed well, with precision ranging from 94. 73% to 95. 03%. However, the proposed ensemble model with multi-head attention outperformed all, achieving an accuracy of 96.35% and the F1-score of 95.76%. These findings highlight the superior ability of the ensemble model to capture complex inter-dependencies between heart disease risk factors, making it a robust tool for clinical prediction.