Detection of Unknown-Unknowns in Human-in-Loop Human-in-Plant Safety Critical Systems

Errors in artificial intelligence (AI)-enabled autonomous systems (AAS) where both the cause and effect are unknown to the human operator at the time they occur are referred to as ‘unknown-unknown’ errors. This paper introduces a methodology for preemptively identifying ‘unknown-unknown’ errors in AAS that arise due to unpredictable human interactions and complex real-world usage scenarios, potentially leading to critical safety incidents through unsafe shifts in operational data distributions. We posit that AAS functioning in human-in-the-loop and human-in-the-plant modes must adhere to established physical laws, even when unknown-unknown errors occur. Our approach employs constructing physics-guided models from operational data, coupled with conformal inference for assessing structural breaks in the underlying model caused by violations of physical laws, thereby facilitating early detection of such errors before unsafe shifts in operational data distribution occur. Validation across diverse contexts–zero-day vulnerabilities in autonomous vehicles, hardware failures in artificial pancreas systems, and design deficiencies in aircraft in Maneuvering Characteristics Augmentation Systems (MCAS)–demonstrates our framework’s efficacy in preempting unsafe data distribution shifts due to unknown-unknowns. This methodology not only advances unknown-unknown error detection in AAS but also sets a new benchmark for integrating physics-guided models and machine learning to ensure system safety.