Automated Identification of Label Errors in Large Electrocardiogram Datasets

Background: Training and testing Deep Neural Networks (DNNs) for automated electrocardiogram (ECG) interpretation requires large datasets. These datasets are commonly extracted at scale from Electronic Health Records (EHRs). Typically, a single physician over-reads the machine generated interpretation as part of standard care. Incorrect interpretation of the ECG occurs frequently, reducing the quality of the labels. Method: We trained a DNN to identify seven ECG rhythms based on morphology; Sinus Rhythm, Junctional Rhythm, Ectopic Atrial Rhythm, Atrial Flutter, Atrial Fibrillation, Ventricular Rhythm and Pacemaker. The DNN was trained on a dataset of 368,202 ECGs taken from a proprietary database. We then applied confident learning techniques using the DNN to identify label errors in the Physionet PTB-XL database, which is publicly available. Results: The confident learning algorithm identified 515 potential rhythm label errors in the 21,837 ECGs in PTB-XL database (2.36%). The labels were sorted by the likelihood of label error based on the self-confidence score, and the top 200 ECGs were manually reviewed. Of these 200 ECGs, 158 were found to be incorrectly labelled (79%). Confident learning successfully corrected the label in 156 cases (78%). The estimated labelling error rate for ECG rhythm in the PTB-XL database is 1.86%.