An Image to 3d Cross-Modal Approach for Real-Time 3D Human Posture Estimation

We present a smartphone RGB camera based system for automatic human 3D posture monitoring driven by our proposed Machine Learning (ML) backbone. Rather than mapping RGB image sequences directly to 3D posture, we learn an intermediate, concise feature space representation of 3D postures. To learn this feature space, we train an auto-encoder learnt on 3D posture sequences. We train a mapping from the image sequence to this concise 3D feature space. This mapping subtracts all unnecessary information present in the image domain and retains only 3D posture information. This elimination helps in dataset independence and increases robustness of our model. The mapped feature space vector is then mapped to 3D posture domain. For estimating 3D human pose, most States-Of-the-Art need either a 3D depth sensor which is expensive or need multiple cameras for stereo-vision. Our proposed cross-modal method uses monocular video from single low-end smartphone camera for estimating human 3D pose in real time ( 5 fps) with high accuracy (less than 1 cm error per joint ~location). We also show our method being effective for posture analysis in yoga and gym exercises.