Randomized testing for Robotic plan execution for autonomous systems

Autonomous underwater vehicles (AUVs) are commonly used for carrying out pre-planned oceanographic surveys, but there is increasing interest in optimizing these surveys by performing onboard re- planning. MBARI has developed an advanced AUV control system, the Teleo Reactive Executive (T-REX) that enables the vehicle to survey areas in more detail if biogeochemical markers indicate the presence of a target feature, and even to follow dynamic ocean phenomena such as fronts. T-REX uses artificial intelligence (AI) techniques in constraint-based temporal planning together with a layered control architecture that allows plans to be generated and executed onboard. One challenge of onboard plan synthesis and execution is that the power of the system to generate different behaviors makes it hard to test in simulation, and failures at sea are costly. We introduce a randomized Monte-Carlo method based test approach that executes hundreds of simulated missions with each mission presenting different inputs to the planner, and checks each output plan for validity. The approach sets environmental parameters to exercise T-REX's domain model, and it is fully configurable. We describe how the Monte-Carlo tester integrates with T-REX, how we have incorporated it into our testing process, and the benefits for system reliability that have resulted. We also highlight our experiences in discovering bugs both in simulation and for science surveys in waters off Northern California.