Detection of Unknown-Unknowns in Human-in-Loop Human-in-Plant Systems Using Physics Guided Process Models

Unknown-unknowns are operational scenarios in artificial intelligence (AI) enabled autonomous systems (AAS) that are not accounted for in the design and validation phase. Their occurrences can be attributed to uncertain human interaction and rigors of regular usage in deployment. The paper illustrates that integrated modeling of human user and AAS from the human-in-the-loop human-in-the-plant (HIL-HIP) perspective can potentially enable detection of unknown unknowns before it results in fatal safety violations. We propose a novel framework for analyzing the operational output characteristics of safety-critical HIL-HIP systems to discover unknown-unknown scenarios and evaluate potential safety hazards. We propose dynamics-induced hybrid recurrent neural networks (DiH-RNN) to mine a physics-guided surrogate model (PGSM) that checks for deviation of the AAS from safety-certified operational characteristics. The PGSM enables early detection of unknown-unknowns based on the physical laws governing the system. We demonstrate the detection of operational changes in an Artificial Pancreas(AP) due to unknown insulin cartridge errors.