Measuring human-robot team effectiveness to determine an appropriate autonomy level

This paper proposes a methodology to measure the effectiveness of a human-robot team as part of an adjustable autonomy system. The effectiveness measure is aimed at determining an appropriate autonomy level prior to the system's deployment. Two competing goals need to be traded off: maximising robot performance while minimising the amount of human input. The relative importance of the two goals depend on the mission priorities and constraints which are taken into account. The proposed methodology is applied to a human-robot communication system developed for task- oriented information exchange. The robot uses a decision- theoretic framework to act autonomously and to decide when to request input from human operators. The latter is achieved by computing the value-of-information an operator is able to provide which is compared to the cost of obtaining the information. For our system, the cost parameter represents the autonomy level to be determined. We demonstrate how an appropriate autonomy level can be found experimentally using a navigation task. In our experiment, the robot navigates through a set of simulated worlds with human input being generated by a software component. The results are used to find appropriate autonomy levels for three example missions and a subsequent user study.