A Lightweight Neural Network Based Human Depth Recovery Method

Human depth recovery is an essential task for tele-immersive video interaction systems with limited bandwidth consumption. This paper is motivated by the first human depth recovery method, i.e., the weighted large margin nearest center (WLMNC) distance based method, which compresses each human depth map into several skeletal block structures by learning a WLMNC distance at the remote stage and then recovers it by a rough-to-fine approach at the local stage. Since the WLMNC distance is equivalent to employing a linear transformation on the human pixels, for human postures with complex self-occlusion, the depth recovery performance of WLMNC is limited. To address the problem, this paper proposes to learn a nonlinear WLMNC distance via a lightweight neural network. Different from traditional classification or clustering problems, the neural network is introduced for storing skeletal block structure information instead of predicting new information. The quantitative experimental results on the VGA-sized benchmark dataset demonstrates that such a slight modification over WLMNC can achieve an inspiring performance improvement: the average depth recovery MAD error is reduced from 3.56cm to 2.77cm and the total running cost is reduced from about 150 seconds to 43 seconds on a laptop with just a little loss on the sampling rate, from 64x to 51x.