Defining Metrics for Autonomous Controllers Assessment

Intelligent systems capabilities are increasing and its application to robotics has become largely popular. Some literature has been produced about autonomous controllers for robotics, demonstrating that they can face a variety of domains. However, we can observe that the experiments performed lack of a common scientific methodology. Generally, isolated case studies are presented, without providing enough details to enable other researchers to replicate or to compare their works with the previous results in the field. The aim of this work is to contribute defining and operationalizing a framework for plan-based controllers assessment. Such effort is supported on a set of generally applicable metrics to entail evaluation of different aspects of robotics controllers. Besides these metrics, we have used OGATE, a domain independent tool that automatically carries on with controller assessments. Then, following a well-defined methodology, we are able to generate reproducible benchmarks, providing objective evidences of the experiments performed. To test our framework, we have used two plan-based controllers on the same robotic problem and platform. Results show that we are able to extract relevant data, allowing comparison of different autonomous controllers.