3D Hand Joint and Grasping Estimation for Teleoperation System

Gesture-based teleoperation is a complex and essential task that enables remote object manipulation. Recent advancements in 3D human hand pose estimation, driven by affordable depth cameras, have proven its aptitude for this task. However, while previous vision-based approaches focus on mapping hand posture to end-effectors, they overlook the interaction between the robot and the object. This leaves a challenge in interpreting these hand joint estimates into practical robotic behaviour. In this paper, we propose a method that leverages the geometric information of the human hand to enable robots to perform human-like grasping and manipulation. Our approach incorporates a pointcloud-based hand joint regressor and the grasping direction analysis (GDA) to control the robot. The joint-wise regressor showed an improved mean joint error of 7.8mm on the MSRA dataset compared to the 8.5mm baseline. We demonstrate that the GDA-based teleoperation can successfully perform real-time robotic manipulator controlling and grasping for various tasks.