How Uncertainty Affects Test Results for Driving Automation

Given the increasing test efforts required for Advanced Driver Assistance Systems and higher driving automation, a variety of test environments has to be leveraged. Ranging from virtual X-in-the-Loop to real-world testing, each test environment entails uncertainties which can be aleatory and/or epistemic. We introduce different sources of uncertainties and discuss their type and allocation to respective test environments. While X-in-the-Loop environments are mainly prone to epistemic uncertainty due to the simplification of simulation models, a substantial part of the uncertainty of real-world testing is aleatory due to nondeterministic effects from driver and environment. Given a nondeterministic test environment, we introduce a method to estimate confidence intervals for test result probabilities based on a multinomial distribution. This method is applied to a scenario for an Advanced Emergency Braking System. It can be shown that based on the confidence intervals, the system meets a predefined minimum performance and test results have sufficient statistical significance. Applying the method to a reduced test set indicates that a lower number of executions can undermine both the argumentation for minimum performance and test result significance.