scenoRITA: Generating Diverse, Fully Mutable, Test Scenarios for Autonomous Vehicle Planning

Autonomous Vehicles (AVs) leverage advanced sensing and networking technologies (e.g., camera, LiDAR, RADAR, GPS, DSRC, 5G, etc.) to enable safe and efficient driving without human drivers. Although still in its infancy, AV technology is becoming increasingly common and could radically transform our transportation system and by extension, our economy and society. As a result, there is tremendous global enthusiasm for research, development, and deployment of AVs, e.g., self-driving taxis and trucks from Waymo and Baidu. The current practice for testing AVs uses virtual tests—where AVs are tested in software simulations—since they offer a more efficient and safer alternative compared to field operational tests. Specifically, search-based approaches are used to find particularly critical situations. These approaches provide an opportunity to automatically generate tests; however, systematically creating
valid
and
effective
tests for AV software remains a major challenge. To address this challenge, we introduce
scenoRITA
, a test generation approach for AVs that uses an evolutionary algorithm with (1) a novel gene representation that allows obstacles to be
fully mutable
, hence, resulting in more reported violations and more diverse scenarios, (2) 5 test oracles to determine both safety and motion sickness-inducing violations and (3) a novel technique to identify and eliminate duplicate tests. Our extensive evaluation shows that
scenoRITA
can produce test scenarios that are more effective in revealing ADS bugs and more diverse in covering different parts of the map compared to other state-of-the-art test generation approaches.