GAN-Uplift: 2D to 3D Uplift with Generative Adversarial Networks

Human pose estimation and prediction has many applications from autonomous vehicles to video games development, animation and security. In many of these instances humans are recorded by video in two dimensions. This two-dimensional representation requires uplifting into three dimensions before being fully utilised. This paper proposes lifting two-dimensional representations of human movement into three-dimensional skeleton representations over a sequence of human action. The proposed approach builds on the work in HP-GAN [1] utilising a generative adversarial network (GAN) with a recurrent neural network generator and a multi-layer fully connected neural network critic. A novel approach adds random noise from a normal distribution as the z dimension for each joint and a custom loss function consisting of the joint position in 3D space and bone length. The proposed algorithm GAN-Uplift successfully uplifts 2D motion sequences into their respective 3D motion sequences, with a sequence mean joint accuracy of 30.9mm and outperforms several state-of-the-art methods and is within 0.4mm of the best state-of-the-art models on the Human3.6M skeleton dataset. In addition, state-of-the-art methods uplift a single pose from a sequence of pose input. GAN-Uplift uplifts a sequence of human poses rather than a single pose.