Classification of 12-lead ECGs: the PhysioNet/Computing in Cardiology Challenge 2020

The PhysioNet/Computing in Cardiology Challenge 2020 focused on the identification of cardiac abnormalities in 12-lead electrocardiogram (ECG) recordings. A total of 66,361 recordings with clinical diagnoses were sourced from five hospital systems in four countries. We shared 43,101 annotated recordings publicly and withheld the remaining recordings for validation and testing. We challenged participants to design working, open-source algorithms for identifying cardiac abnormalities in 12-lead ECG recordings. We sourced data from several institutions with different demographics, required participants to submit code for training their models, and proposed a novel evaluation metric that awards partial credit for misclassified cardiac abnormalities with low risks or similar outcomes as the actual abnormalities. These innovations encouraged the development of generalizable, reproducible, and clinically relevant algorithms. A total of 217 teams submitted 1,395 algorithms during the Challenge, representing a diversity of approaches from both academia and industry for identifying cardiac abnormalities. Algorithms performed similarly on the validation and test data with a drop of roughly 10% in performance on the completely hidden data, illustrating the difficulty of adapting algorithms to novel data.