Scenario-Based Methods for Machine Learning Assurance

Synthetic data can support the design assurance of machine learning enabled components in aviation at a relatively low cost compared to in-field data collection campaigns and flight tests. In this paper, we study the methods for generating synthetic datasets using operational design domain descriptions and scenarios - informal or formal descriptions of the environment and the actions of objects over time. We propose a workflow where high-level scenario descriptions are parameterized with the operational envelope data of the machine learning system, and then concrete scenarios can be sampled from this parameter space. Concrete scenarios can be used to set up physics-based models to obtain synthetic data elements, such as images, via simulation. This data can then be used as part of either training or test dataset to support, respectively, training and testing of the model. We implement the scenario-based data generation workflow using state-of-the-art optimization and simulation tools, and demonstrate the results on a realistic case study for maritime search and rescue with unmanned aerial vehicles.