Finding system-level failures in flight-critical software systems

NextGen civil aviation capabilities depend heavily on systems-of-systems that interact to produce unexpected behavior. To validate system- level goals, the prototype software for the system needs to be coupled with high-fidelity physics simulations of the hardware and the environment. The associated state spaces are massive and contain nonlinear and stochastic elements (like weather models) that formal methods do not currently treat easily. Testing methods that use machine learning have been shown to aid in exploring such systems. In particular, blind source selection methods can aid in intelligently reducing the state space, and feature selection methods can be used to choose input space features leading to desired or undesired output space behavior. When the global state space is too large to solve explicitly, we can use machine learning and statistical techniques to build models of the system. These simpler models enable us to predict behavior in the high-fidelity simulation, then adaptively refine the models as we test our predictions. This paper uses model-based testing to exercise a new air traffic control concept. The concept is implemented in software that helps controllers detect and resolve short-term conflicts between aircraft in the terminal airspace. The rules that determine whether or not aircraft are sufficiently separated within 40 miles of the terminal depend on aircraft weight, the flight rules, the type of approach, and whether the aircraft is arriving or departing. We show that model-based testing automates the process of feature selection and state-space reduction, enabling the analyst to quickly validate expected behavior and explore anomalies.