Automatic Test and Evaluation of Autonomous Systems

Physical test and evaluation (T&E) of autonomous systems in actual settings is resource intensive and time exhaustive. Simulation-based testing, however, can reduce the testing cost and time, allowing for testing the whole operation envelope through a massive set of scenarios. In this research, a novel automatic simulation-based testing is proposed that simultaneously amalgamates the knowledge of domain experts and the operating and environmental parameters of autonomous systems. The proposed method employs fuzzy logic to replace the exhaustive activities of the tester and leverages the tester’s role to review root-cause reasoning reports about the System Under Test (SUT) and focus only on the inspection of flagged scenarios rather than inspecting all instants of all scenarios. The proposed method uses Type-2 fuzzy logic to provide more robust handling of data uncertainties involved in the testing process. An integrated configurable software tool and a user-friendly Graphical User Interface (GUI) are developed that allow for testing a single and/or batches of large number scenarios to generate test reports along with the root-cause analysis. The developed software tool has been successfully applied for testing the perception system of Unmanned Aerial Systems (UAS) in the Collaborative Unmanned Systems Technology Demonstrator (CUSTD).