Auto-Scenario Generator for Autonomous Vehicle Safety: Multi-Modal Attention-Based Image Captioning Model Using Digital Twin Data

The field of autonomous vehicles (AVs) scenarios has become a more vital process to secure safety. However, the existing scenario approaches to assess AVs have three limitations: low representativeness, low diversity, and low efficiency. Since the real AV driving data cannot be accessed publicly, the scenarios based on low-dimensional data are inadequate to secure representativeness and diversity. Also, generating myriad scenarios using human resources (numerous experts’ input) is inefficient and often lacks consistency. Recognizing these issues, we present a novel approach for generating scenarios of AVs safety. Our approach emphasizes process efficiency in generating scenarios while ensuring diversity and representativeness. We devise a multi-modal image captioning model, referred to as Auto Scenario Generator (Auto-SG), which automatically generates accident scenarios using digital twin data. This model consists of inception-v3, attention mechanism, and GRU. Using SCANeR studio, we also implement various AV accident situations and extract the images for model training. We identify the optimal Auto-SG model through extensive experimentation, enabling the model to generate captions similar to real captions. To evaluate the model, we use BLEU@N and ROUGE-L metrics. Our model achieves approximately 80 to higher scores, demonstrating that the captions of AV accident scenarios are generated correctly. Among the results, we exemplify six top-accuracy accidents as to formalized functional scenarios for assessing AV safety. The scenarios present two intersections and four road-related accidents, including the behaviors of lane changing and turning right. Finally, we suggest the qualitative criterion of ’efficiency’ to evaluate these scenarios, which can consider investments of human resources and consistency of scenarios. We believe our model can help us generate scenarios more efficiently while ensuring diversity and representativeness.