Perception of Demonstration for Automatic Programing of Robotic Assembly: Framework, Algorithm, and Validation

In this paper, we focus on developing an intelligent perception system that can generate the task program for the robot to realize programming by demonstration (PBD) in industrial assembly tasks. The core problem of the system is to understand the semantics of the parts and skills in the visually observed demonstration. We present a probabilistic framework to state this problem as the joint inference of actions in the skill, as well as the parts poses and relations. To estimate the former, a multimodal information fusion action recognition algorithm is developed that makes use of the prior knowledge in the assembly task, so that real-time performance is achieved. While for the latter, an expectation maximization based algorithm is designed to generate solutions meeting the high accuracy required by the assembly. The perception system is validated on comprehensive simulation data and the real assembly tasks by committing the generated program to an ABB industrial robot. The satisfactory results indicate the effectiveness and feasibility of the system, finally verifying our hypothesis that PBD can be applied in assembly tasks to achieve the “teacher” and “student” scenario with the proposed perception system.