A Probabilistic Approach to Benchmarking and Performance Evaluation of Robot Systems

Problem benchmarks are used in experimental science as a reference against which results of experiments using distinct approaches to solve the problem are compared and evaluated in relative terms. In Robotics, just formulating a general performance assessment problem is difficult per se, as robot systems are composed of very diverse subsystems (e.g., localisation, human-robot interaction, task planning, motion planning). This paper introduces a probabilistic approach to benchmarking and evaluating performance of robot systems, which uses probability theory as the common language to quantify the performance of distinct functionalities of a robot system and their impact on the performance of a task carried out by that system. The approach can be used to analyse the performance of a task plan from the performances if its composing functionalities, or to (re)plan when a performance degradation in functionality is predicted to cause performance degradation of the task plan beyond acceptable limits.