Joint Hand-Object 3D Reconstruction From Monocular Image Based on Fused Visual Cues and Pose Prior

Joint reconstruction of hands and manipulated objects from a monocular image is important perception technique which has recently achieved impressive progress. Compared to reconstructing hands and objects from temporal sequences, hand and object reconstruction from a single monocular image does not require continuous image frames and can be performed with just one camera, which is applicable to a wider range of scenarios. In existing methods, signed distance field is widely used due to its excellent geometric representation ability. However, visual cues and pose prior information are not fully explored in signed distance field estimation. Therefore, this paper proposes a new hand-object reconstruction framework to fuse implicit representations with more representative prior information, in which other prior information and visual cues are fully explored. First, when signed distance value of a 3D point is queried, both global features and local features from 2D image plane transformed from the 3D space are integrated. Second, the anchor-based mechanism is introduced into the hand keypoints prediction, which provides a more accurate pose prior. In the comparison with the state-of-the-art model, the experiment results on common benchmarks demonstrated that the proposed method had good performance and achieved a shape error about $0.106~cm^{2}$ on Obman and $0.412~cm^{2}$ on DexYCB. Especially on the Obman dataset, the hand chamfer distance $CD_{h}$ reduces 5.4%, the object chamfer distance $CD_{o}$ reduces 27.1%, the hand keypoints error $E_{h}$ reduces 5.7% and the object center error $E_{o}$ reduces 9.0%.