Human-Robot Sign Language Motion Retargeting from Videos

Motion retargeting is an effective way to quickly equip robots with sign language capabilities and alleviate the difficulties of communicating with the hearing impaired. However, current sign language motion retargeting methods rely heavily on precise motion capture data, which is expensive and difficult to acquire, limiting the amount and variety of data. In comparison, sign language videos are easier to obtain. In this paper, we propose a general framework for sign language motion retargeting from videos to address this problem. By combining human pose estimation, we extend sign language motion retargeting to video input. Furthermore, we introduce spatial-temporal graphs to capture the intricacies of sign language movements. We perform experiments on sign language motion retargeting on ABB’s YuMi dual-arm collaborative robot in the simulation environment. The experimental results demonstrate the effectiveness of our proposed method in preserving motion similarity while adhering to robot constraints.