PeMapNet: Action Recognition from Depth Videos Using Pyramid Energy Maps on Neural Networks

We propose an integrated approach to human action recognition from a depth video. The two major contributions of this approach are a novel feature descriptor for depth videos and the corresponding deep learning neural network structures. In this paper, we first present pyramid energy Maps (PeMaps) as the feature descriptor for a sequence of frames in a depth video. The pyramid structure is able to present the history of an action. Furthermore, PeMaps uses the levels of energy to carry the spatial dynamics of actions in a depth video. We then design PeMapNet that applies convolution neural networks and bidirectional long-short term memory (BLSTM) recurrent neural networks to PeMaps for action recognition. We evaluate our approach on three challenging datasets including MSR-Action3D, UTKinect-Action3D and MSR-Gesture3D. The experimental results demonstrate that our approach obtained higher accuracy than most of the existing methods and advance in efficiency.