Graph-based semantic planning for adaptive human-robot-collaboration in assemble-to-order scenarios

Assemble-to-Order (ATO) has become a popular production strategy for the increasing demand for mass-customized manufacturing. In order to facilitate a flexible and automated Human-Robot-Collaboration system for ATO, we propose a Learning from Demonstration (LfD) framework based on 2D videos in this paper. We initially combine temporal hand motions with spatial hand-object interactions to detect assembly actions. Therefore, an assembly graph can be constructed using classified action sequences. Compared to previous studies on task planning for robots, our graph-based semantic planner can directly learn the demonstrated task structure and thus produce more detailed assistive robot actions for more effective collaboration. We validate our approach by applying it to a real-world ATO problem. The results demonstrated that our proposed system can produce actions adaptively in response to varying human action sequences, as well as guide human assembly when the robot is not involved. Our approach also shows generalizability to unseen human action sequences.