Performance measure for the evaluation of mobile robot autonomy

Measuring the performance of robots, and more precisely of autonomous robots dealing with complex and changing environment, is a long standing issue. There is today no satisfactory known metric to measure quantitatively and precisely how a robot system satisfies accurately the specification expectations, neither do we know how to qualify and delineate in general the operational domains of the robot. This paper presents an approach aiming at qualifying robot autonomy. We present a method based on the measure of robot performance in achieving a given task related to environment complexity. In order to test a large number of tasks and environment configurations, we have designed a robot simulator that allows the use of the same algorithms implemented on the real robot and in simulation. To illustrate the work, we present results based on hundreds of simulations of a navigation mission and compare robot performance when algorithm parameters change