Depth-Hand: 3D Hand Keypoint Detection With Dense Depth Estimation

Hand pose is important to various applications and depth information is crucial for a reliable 3D keypoint detection. However, scopes of methods that rely on active depth cameras are limited by the power, volume and illumination. To explore a wider application range, this letter proposes a multi-task method that can detect 3D hand keypoints while estimating dense depth maps from stereo infrared inputs. Based on the inherent encoding-decoding relation of the depth estimation and hand keypoint detection, the proposed network is built with shared intermediate features and separate task branches. To achieve an end-to-end estimation, the hand region is automatically cropped from the depth map. Due to a lack of datasets, a two-step fusion training approach is designed following the transfer learning theory with a self-supervision loss. The proposed method is evaluated under stereo depth datasets and 3D hand keypoints datasets respectively. Meanwhile, a small dataset is also built to test the the overall model performance. Experimental results prove that the proposed method is capable of providing satisfying depth maps along with convincing 3D hand keypoints.