External Validation Complexities: A Comparative Study of Late-onset Sepsis Prediction Models Across Multiple Clinical Environments

Objective: Neonatal late-onset sepsis (LOS) is a life-threatening condition in preterm infants in neonatal intensive care units (NICUs), with early detection being crucial for improving outcomes. Despite advancements in data-driven prediction models, their generalizability remains uncertain due to a lack of independent validation, particularly on national and international scales. This study evaluates the performance of two LOS prediction models on multiple validation datasets to assess their reliability for clinical implementation. Methods: Two models were validated: (1) a multi-channel feature-based extreme gradient boosting model (MC-XGB) and (2) a deep neural network using raw RR intervals (RR-DNN). Validation was conducted on three NICU datasets: an internal dataset (68 LOS, 100 controls) from the model-development hospital in the Netherlands, a national external dataset (20 LOS, 20 controls) from another Dutch hospital, and an international external dataset (17 LOS, 17 controls) from a U.S. hospital. Model performance was assessed using the area under the receiver operating characteristic curve (AUC) across multiple prediction time windows, with an hourly risk analysis. Results: Both models achieved a peak AUC of 0.82 in the internal dataset, their predictive performance demonstrates variable declines in external datasets. The respective AUCs for RR-DNN and MC-XGB were 0.80 and 0.72 in the national dataset, and 0.69 and 0.60 in the international dataset. This may result from variations in clinical practices, patient demographics, and monitoring technologies. Conclusion: Model performance declined in external validations, highlighting the challenges of implementing predictive models across diverse clinical settings. Significance: This study emphasizes the need for standardized guidelines and improved data sharing to enhance model development and facilitate reliable integration into NICU workflow for improved LOS management.