A Metamorphic Testing Framework and Toolkit for Modular Automated Driving Systems

Autonomous vehicles (AV), at their highest potentials, will provide a greater mobility, an increased traffic efficiency and, more importantly, safer trips for millions of people every day. While assuring their safety and reliability is, thus, of great importance, it is also a huge challenge. Metamorphic testing (MT) has been shown to be one of the most successful testing techniques to validate automated driving systems (ADS) underpinning the AV. Having said that, there are still lots of rooms for further improving the ADS testing with MT. On one hand, the non-determinism in ADS’ behaviors poses great challenges for precisely judging their correctness. On the other hand, the testing scenarios used in the existing studies are still not very much complex for mimicking various realistic traffic conditions. In this study, we propose a new framework which takes into account the hypothesis testing to provide a more solid way for judging the non-deterministic behaviors of test outcomes. On top of that, we develop a new toolkit to implement more complex and realistic ADS testing scenarios. To demonstrate its practicability, we design complex traffic scenarios and pay attention to examine the ADS’ behaviors in non-collision cases which are often unable to be detected by conventional testing methods. It is then applied to test Autoware, a state-of-the-art modular ADS using the Carla simulator. An analysis of results with the Mann-Whitney-Wilcoxon test and Cohen’s d values reveals a large number of consistencies and reliability issues of Autoware. The findings highlight the flexibility and capability of our MT-based framework in validating the AV using a nondeterministic measure and realistic scenarios that can work in the absence of ground truth datasets.