Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning

We present a method for reconstructing accurate and consistent 3D hands from a monocular video. We observe that the detected 2D hand keypoints and the image texture provide important cues about the geometry and texture of the 3D hand, which can reduce or even eliminate the requirement on 3D hand annotation. Accordingly, in this work, we propose $\mathrm{{S}^{2}HAND}$, a self-supervised 3D hand reconstruction model, that can jointly estimate pose, shape, texture, and the camera viewpoint from a single RGB input through the supervision of easily accessible 2D detected keypoints. We leverage the continuous hand motion information contained in the unlabeled video data and explore $\mathrm{{S}^{2}HAND(V)}$, which uses a set of weights shared $\mathrm{{S}^{2}HAND}$ to process each frame and exploits additional motion, texture, and shape consistency constrains to obtain more accurate hand poses, and more consistent shapes and textures. Experiments on benchmark datasets demonstrate that our self-supervised method produces comparable hand reconstruction performance compared with the recent full-supervised methods in single-frame as input setup, and notably improves the reconstruction accuracy and consistency when using the video training data.

I. Introduction

Hands play a central role in the interaction between humans and the environment, from physical contact and grasping to daily communications via hand gesture. Learning 3D hand reconstruction is the preliminary step for many computer vision applications such as augmented reality [1], sign language translation [2], [3], and human-computer interaction [4], [5], [6]. However, due to diverse hand configurations and interaction with the environment, 3D hand reconstruction remains a challenging problem, especially when the task relies on monocular data as input.