Automated Testing With Temporal Logic Specifications for Robotic Controllers Using Adaptive Experiment Design

Many robot control scenarios involve assessing system robustness against a task specification. If either the controller or environment are composed of “black-box” components with unknown dynamics, we cannot rely on formal verification to assess our system. Assessing robustness via exhaustive testing is also often infeasible if the number of possible environments is large compared to experiment cost. Given limited budget, we provide a method to choose experiment inputs which accurately reflect how robustly a system satisfies a given specification across the domain. By combining signal temporal logic metrics with adaptive experiment design, our method chooses each experiment by incrementally constructing a surrogate model of the specification robustness. This model then chooses experiments in areas of either high prediction error or high uncertainty. Our evaluation shows how this adaptive experiment design results in sample-efficient descriptions of system robustness. Further, we show how to use the constructed surrogate model to assess the behaviour of a data-driven control system under domain shift.