SPMHand: Segmentation-Guided Progressive Multi-Path 3D Hand Pose and Shape Estimation

Hand pose and shape estimation plays an important role in numerous applications. A cost-effective and practical-friendly approach is to perform accurate hand estimation from a single RGB image, but this task is challenging due to ubiquitous hand self-occlusion and hand-object interaction occlusions. In this paper, we propose a novel SPMHand network to alleviate the effect of occlusions, inspired by the process that humans infer the whole hand when the hand is occluded. The proposed SPMHand consists of two main modules to generate hand segmentations as guidance and conduct hand regressions in a progressive multi-path manner. The segmentation-guided deocclusion module enables the network to “see” the occluded hand by inferring the whole hand segmentation. Specifically, the visible hand segmentation is first obtained and then a hand morphology attention block is introduced to infer the whole hand segmentation by fusing the visible information with the learned hand features. The progressive multi-path regression module is designed to gradually regress the fine hand with intermediate supervisions. Features from deep to shallow are utilized for the hand regressions from coarse to decent. Subsequently, the structure feature, joint heatmaps and segmentations that provide guidance for deocclusion are embedded and fused for the final fine hand regression. Experiments on four challenging datasets illustrate that the proposed SPMHand outperforms the state-of-the-arts in both real-world and synthetic scenes, especially in the present of severe hand-object occlusions.