Key Spatio-Temporal Energy Information Mapping for Action Recognition

Nowadays, human action recognition (HAR) based on depth video sequences has become a popular research topic. However, some existing feature map representations suffer from the issue of missing temporal information and cannot completely represent the spatio-temporal information. Moreover, the redundant information is not effectively removed, and the key information cannot be highlighted. To address these issues, we propose an effective method called key spatio-temporal energy information mapping (KSTEIM). First, we construct a human motion energy field to strengthen the distribution of motion energy. Next, we present a new key frame algorithm that removes the redundant frames based on redundancy values calculated from image differences to obtain keyframe sequences. Then, each key frame is projected onto three orthogonal axes to obtain three 1-D motion energy lists. The energy lists are concatenated in temporal sequence under each projection axis to form KSTEIM. Then, we extract histogram of oriented gradients (HOG) from KSTEIM and make KSTEIM-HOG sparser through multi-kernel subspace representation enhancement (MKSRE). With only positive sequence actions, our method achieves a recognition rate of 93.41% on Microsoft research Action3D (MSR-Action3D), and 90.93% on the University of Texas at Dallas Multimodal Human Action Dataset (UTD-MHAD). It indicates that the performance of our method is no weaker than most existing methods.