Generating Executable Test Scenarios from Autonomous Vehicle Disengagements Using Natural Language Processing

With the emergence of autonomous vehicles comes requirements on adequate and rigorous testing techniques, particularly as systems continuously adapt to changing environments. Scenario-based, simulated testing is one approach that has received attention, where deriving relevant scenarios from various sources is still a challenge. We therefore explore creating executable test scenarios from textual disengagement reports, collected from autonomous vehicle test drives, by DMV California. We mined information from 183 182 disengagements, using NLP techniques and developed a tool to output scenarios in OpenScenario format. The data quality of the reports was substandard, resulting in only 36 disengagements be useful and half of the generated scenarios were correctly reconstructed. However, the NLP approach was effective and may be used for other data sets. Further work includes working with more and better data sources and advancing the scenario generation.