Towards Detecting Unintended Behaviors in Machine Learning Algorithms

Software verification is essential to ensure that a system's implementation conforms to its intended functionalities and safety requirements. Traditional aviation software methods, proposed by DO-178C, emphasize coverage metrics like MCDC to verify that all behaviors of the code are as intended. However, these methods fail when applied to machine learning (ML) software, where behaviors are influenced not just by the source code but significantly by the training data and the algorithms used. This paper introduces a novel statistical testing algorithm tailored for ML applications, designed to detect potential un-intended behaviors by analyzing unseen data points within the operational design domain (ODD). The utility of this algorithm is validated through case studies on polynomial regression models, a neural network for cabin pressure control, and LSTM models trained with aerospace simulation data. These studies highlight the algorithm's effectiveness in uncovering unintended behaviors across various ML architectures and data environments.