Integrated planning for telepresence with time delays

Teleoperation of remote robotic systems over time delays in the range of 2-10 seconds poses a unique set of challenges. In the context of a supervisory control system for the JSC Robonaut humanoid robot, we have developed an "intelligent assistant" that integrates an artificial intelligence planner (JSH0P2) with execution monitoring of the state of both the human supervisor and the remote robot. The assistant reasons simultaneously about the world state on both sides of the time delay, which represents a novel application of this technology. The purpose of the assistant is to provide advice to the human supervisor about current and future activities, derived from a sequence of high-level goals to be achieved. To do this, the assistant must simultaneously monitor and react to various data sources, including actions taken by the supervisor who is issuing commands to the robot (e.g. with a data glove), actions taken by the robot, and the environment of the robot, both as currently perceived over the time delay, along with the current sequence of goals. We have developed a "leader/follower" software architecture to handle the dual time-shifted streams of execution feedback. In this paper we describe the integrated planner and its executive, and how it operates in normal and anomaly situations.