BERT-Based Deep Learning for Classification of IgA Nephropathy, Diabetes, and Hypertension from Clinical Notes

Chronic kidney disease affects over 800 million people globally and has many etiologies, including hypertension, diabetes, and IgA nephropathy. Effective management relies on precise medical documentation to monitor disease progression and treatment efficacy. Artificial intelligence, specifically BERT-based models, has shown promise in enhancing disease classification from clinical notes. In Ireland, clinical notes from hemodialysis and nephrology outpatients are stored in the Kidney Disease Clinical Patient Management System (KDCPMS). This study uses DistilBERT with 2,947 unstructured clinical notes from the KDCPMS to classify individual patient notes as stating that the patient has or does not have IgA Nephropathy, Diabetes, and Hypertension. The training pipeline involved data extraction, anonymization, pre-processing, annotation, and model training. On the test set the IgA Nephropathy classifier achieved high precision (99%) and recall (100%), effectively identifying positive cases with minimal errors. The Diabetes classifier had precision of 76% and recall of 90%. The Hypertension classifier had high precision (91%) and medium recall (72%). The performance of the classifiers was further evaluated against a clinical validation dataset of 112 patients created through a full chart review by clinicians. The IgA, Diabetes and Hypertension classifiers had precision of 83%, 61% and 73% respectively and recall of 83%, 73% and 83% respectively. The results highlight the models' performance on the different classification problems in both the test set and a more clinically focused dataset. The results emphasize the importance of ongoing refinement and expert collaboration to enhance clinical relevance and applicability.