A Novel Skeleton Spatial Pyramid Model for Skeleton-based Action Recognition

With the development of computer science and the rise of deep learning technologies, the skeleton-based action recognition dataset has become larger and larger, which has pushed experts and scholars in the field of action recognition to seek more efficient and accurate algorithms. Considering the most critical factors in the task of action recognition are the intra-frame representation of the joints of a skeleton and the inter-frame representation of the skeleton sequence, we propose a novel skeleton spatial pyramid model (S-SPM). The spatial information of different levels is gradually weighted and aggregated, which effectively models the spatial features of the skeleton sequence. Then the spatio-temporal feature representation based on the skeleton spatial pyramid model is proposed to model the temporal information to obtain deep spatio-temporal feature. Finally, the spatio-temporal feature is fed into the convolutional neural network (CNN) to effectively recognize the actions. The experimental results of the proposed algorithm in the NTU RGB+D dataset show that the S-SPM can improve the accuracies for skeleton-based action recognition.