ActionFormer: Pose-based Action Recognition via Transformer and CNN

Recently, human pose estimation and human skeleton capture system developed rapidly and raised more and more attention to pose-based action recognition. Deep learning algorithms like CNN and RNN have been applied to many tasks for the past few years. Although they have showed promising prospects, these methods lack the ability to take full advantage of spatio-temporal relationship information between the skeletons in the behavior explicitly, and the accuracy of action recognition is not easy to improve. We put forward a new and effective CNN-Transformer model able to extract spatial-temporal information for human pose-based action recognition to address these problems. Firstly, we extract the pose features from the original video. Secondly, the pose features are processed by action modeling module. Then, we feed the spatio-temporal information of the skeleton points generated by the modeling into CNN and Transformer modules respectively, in the form of relative position and relative speed. Finally, the output information obtained from the two modules is fused. What is more, we compare the model with the previous advanced methods and carry out comparative experiments on SUB-JHMDB, SYSU-3D and NTU RGB+D datasets. It is obviously that the model significantly improve the action recognition performance through the experimental results.