Human Motion Enhancement Using Nonlinear Kalman Filter Assisted Convolutional Autoencoders

Human motion analysis is integral to many applications ranging from biomedicine to surveillance and 3D animation. The availability of consumer RGB-D sensors makes motion capture (Mocap) more prevalent and accessible in our daily lives. However, depth-based Mocap (D-Mocap) suffers significant errors and noise due to the limitation of depth sensing, self-occlusion, and many other problems. We present a novel filter-assisted deep learning approach to improve low-quality human motion data (e.g., D-Mocap) by taking advantage of the recent progress in both deep learning and nonlinear Kalman filtering. At the heart of this method is a learned motion manifold through the use of a convolutional autoencoder trained on high-quality, rich-variety CMU Mocap data which is used to recover valid human motion from corrupted input. Furthermore, the Tobit Kalman filter (TKF), proposed to handle censored measurements, is used to assist the autoencoder with more kinematic and dynamic constraints. Two structural paradigms are investigated to handle different kinds of data errors by maximizing the synergy between the two integral parts in this work. The experimental results on both simulated and real-world human motion data demonstrate the effectiveness and robustness of the proposed methods to improve the quality of noisy and erroneous Mocap data.