Using Decision Support in Human-in-the-Loop Experimental Design Toward Building Trustworthy Autonomous Systems

Experimental design of autonomous systems involves defining experimental inputs to maximize the experimenter’s information gained, minimize costs, and balance risk. This effectively leads to improved understanding and trustworthiness, which are necessary for deployment in realworld settings. Since experimental design is inherently a human-in-the-loop, sequential decision making problem, and decisions are being made about complex systems, an investigation into decision-making quality and decision-supporting methods is warranted. In this work, we investigate a decision support system (DSS) to augment the human’s experimental design decision making abilities, and conduct an exploratory user study to investigate the potential for decision support. Our findings show that experimenters, including experienced field roboticists, make suboptimal decisions and mistakes during the experimental design process, which suggests robotics research could benefit from DSSs. Our proposed DSS shows promise in some select aspects of experimental design, including helping to reduce suboptimal decisions, and participants in the user study reported favorable opinions of using such a system, including a sense of usefulness and lack of burden. The broader implication of this work is the identification of decision support in experimental design as one way to help bridge the gap between academia and industry by way of accelerated, informative experimentation and increased system explainability.