Motion energy guided multi-scale heterogeneous features for 3D action recognition

This paper is to address the problem of human action recognition in depth sequences. The actions with various speeds and shared sub-actions make the recognition challenging. A new feature set, consisting of two heterogeneous features are proposed to address this challenge. Specifically, we propose an adaptive normalized action motion energy based on the depth video. Guided by this multi-scale energy vector, depth sequence and skeleton pose sequence are divided respectively into two sets of subsequences with multiple scales (i.e., multi-scale sub-actions). Then in depth modality, based on the depth sub-sequence, Depth Motion Maps (DMMs) based Histogram Oriented Gradient (HOG) features are employed to capture the shape information and motion cues. In skeleton modality, based on the pose sub-sequence, pose dynamics using skeleton information are extracted. In order to obtain discriminative and compact representation, the Collaborative Representation (CR) learning scheme based classifier is adopted. Experiments on two datasets show the effectiveness of the proposed method.