Robot Symbolic Motion Planning and Task Execution Based on Mixed Reality Operation

With the increasing demand for human–robot collaboration (HRC), intuitive interfaces are essential to connect humans and robots. A promising approach is the use of mixed reality (MR) to enhance spatial understanding through virtual and augmented reality. In this paper, we propose a novel HRC system that extracts human handling procedures and generates concrete motion plans for the robot. The user, wearing an MR device, interacts with virtual objects in the MR space using natural hand motions. These motions and resulting state transitions are abstracted into a symbolic semi-order motion planner represented by the reachability graph (RG). Using the RG, an autonomous behavior tree is generated, considering the robot’s task environment, and the concrete motion plan is executed by the robot. This system allows the robot to take a more flexible approach to user instructions than conventional MR-HRC systems. Moreover, this system translates human orders into plans that are independent of a specific robot, demonstrating considerable development potential.