Algorithms for Detection of Anomalies in Data From Clinical Studies and Medical Registries

Anomaly detection is a burgeoning trend in data quality, underpinned by the fundamental principles of honesty and truthfulness in scientific research. Adherence to these principles is critical for advancing science and ensuring public trust in scientific findings. Deviations from these principles can lead to scientific misconduct or fraud, with potentially severe consequences, particularly in clinical research where human lives may be at stake. This work introduces an implemented algorithm for anomaly detection, leveraging basic machine learning concepts such as training on original datasets and testing on new datasets. Two training methods, resubstitution, and external validation (hold-out), were employed. To assess classification accuracy, the input dataset was deliberately augmented with anomalous records, and the detection results were compared against a vector of artificially generated anomalies. Prior studies may have overlooked factors such as varying anomaly occurrence rates and the need for manual threshold adjustments, which our algorithm addresses. By addressing these limitations, the algorithm contributes to the advancement of anomaly detection methodologies in clinical research and beyond.