A Reliability Evaluation Methodology for X-in-the-Loop Simulation in Autonomous Vehicle Systems

X-in-the-loop simulation (XILS) is a method to validate automated vehicle systems that combines several simulation techniques. XILS combines real hardware, software, and virtual models to test the functionality of autonomous driving systems. XILS is particularly effective because it enables repeatable and reproducible testing across diverse scenarios and can comprehensively evaluate system performance. While researchers have had great success implementing XILS platforms that closely approximate real-world vehicle testing environments, the performance of vehicles in XILS testing may still differ from what is observed in real-world testing, especially with respect to the reliability of the test results. In other words, better methods are still needed to evaluate the reliability of XILS platforms. In this paper, we propose such a method. The new approach includes scenario definition, key test parameter definition, and procedures to evaluate consistency and correlation based on statistical and mathematical comparisons between the datasets from XILS tests and real-world tests. This is done from both a parameter and scenario perspective. Furthermore, we introduce and examine reliability evaluation criteria for XILS that are based on empirical findings from repeated tests. In order to determine the effectiveness of the proposed methodology, we used it to evaluate the reliability of a vehicle-in-the-loop simulation, which is a specific approach within XILS, in the form of a case study. Ultimately, this paper analyzes the factors that impact the reliability of simulation-based validation for automated vehicle systems and provides guidance for improving overall reliability.