DistilBERT Meets Diagnosis: Multi-Class Disease Classification from Noisy Medical Transcriptions

Medical transcriptions, a rich yet underutilized source of clinical information, offer significant potential for supporting computer-aided diagnosis. However, this data is often unstructured, error-prone, and exhibits severe class imbalance—posing challenges for automated disease classification. In our research work, we present a novel framework called "DistilBERT" for multi-class disease classification from medical transcription text, leveraging machine learning and deep learning models. The DistilBERT includes noise reduction through preprocessing, followed by class balancing using synthetic oversampling techniques such as SMOTE, ADASYN, and SMOTE-Tomek. To better understand data distribution and class separability, we employed dimensionality reduction methods such as PCA, t-SNE, and UMAP. We benchmark a range of models—Logistic Regression and Support Vector Machines on the classical side, and CNN, BiLSTM, Transformer-based models, and DistilBERT on the DL front. Among these, DistilBERT achieves the highest performance, with accuracy of 87.6% and an F1 score of 0.871. To enhance interpretability, we apply post hoc explainability tools like LIME and SHAP, revealing that model predictions are influenced by clinically relevant terms. Our results highlighted the value of integrating data balancing, deep learning, and explainability in building reliable and interpretable systems for medical text classification.