Towards Assurance Evaluation of Autonomous Systems

Due to the probabilistic and uncertain nature of learning-enabled autonomous systems, new assurance technologies appropriate for such systems are becoming critical for their acceptance. Runtime monitors are among the key assurance technologies to address these challenges. In the DARPA Assured Autonomy program, our Boeing team is performing autonomous platform integration and assurance technology evaluation. In this paper, we present our preliminary evaluation results for runtime monitor technologies, which were developed by our three partner teams during Phase I for learning-enabled autonomous systems. The evaluation was completed using a flight simulator and real platforms. In the evaluation, the demonstrated assurance technologies showed significant promise in addressing the assurance shortfall currently facing learning enabled cyber physical systems, and the evaluation approach defined here provides a solid starting point for evaluations of the maturing assurance technologies in the future phases of the DARPA Assured Autonomy project.