Learning from Human Hand Demonstration for Wire Harness Grasping

In recent years, the automation of bin picking in factory environments has made significant strides. The automation of rigid objects, such as metal components, has been successfully implemented by leveraging 3D data. However, for deformable objects like wire harnesses, where the object’s pose is uncertain, practical implementation is challenging. In most cases, manual intervention remains predominant. To overcome this problem, we propose a system wherein human operators teach a robot wire harness grasping actions through hand demonstrations. The process involves capturing human grasping of the wire harness and instructing the robot based on RGB-D images to learn the human grasped location and grasping posture. We notice that human tends to grasp specific regions with characteristic structures of wire harnesses. In order to learn such information, we propose a method to build a dataset for neural network training with few shot images. We form the problem as instance segmentation and augmentation of the training dataset is achieved by overlaying wire harness images onto various background scenes. Next, the obtained point cloud of grasping locations is aligned with the point cloud from the demonstration instances through point cloud registration. Using such information, the robot transfers the wire harness grasping pose during the demonstration to the current scenes. We evaluated the accuracy of grasping location segmentation and the success rate of wire harness grasping in real experiments.