Improving Evaluation Methodology for Autonomous Surface Vessel COLREGS Compliance

This article presents methodology for evaluating the performance of black box autonomous surface vessel (ASV) software, with particular focus placed on determining COLREGS compliance in the presence of competing priorities. The approach utilizes adaptive sampling and unsupervised clustering to automatically identify failure modes and performance boundaries in the system under test. Analysis of the evaluation methodology applied to a candidate ASV is presented in a high-dimensional testing space. Results show that conventional methods for testing ASV software are incomplete for the purpose of evaluating the full range of system behavior.