2-D Skeleton-Based Action Recognition via Two-Branch Stacked LSTM-RNNs

Action recognition in video sequences is an interesting field for many computer vision applications, including behavior analysis, event recognition, and video surveillance. In this article, a method based on 2D skeleton and two-branch stacked Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) cells is proposed. Unlike 3D skeletons, usually generated by RGB-D cameras, the 2D skeletons adopted in this article are reconstructed starting from RGB video streams, therefore allowing the use of the proposed approach in both indoor and outdoor environments. Moreover, any case of missing skeletal data is managed by exploiting 3D-Convolutional Neural Networks (3D-CNNs). Comparative experiments with several key works on KTH and Weizmann datasets show that the method described in this paper outperforms the current state-of-the-art. Additional experiments on UCF Sports and IXMAS datasets demonstrate the effectiveness of our method in the presence of noisy data and perspective changes, respectively. Further investigations on UCF Sports, HMDB51, UCF101, and Kinetics400 highlight how the combination between the proposed two-branch stacked LSTM and the 3D-CNN-based network can manage missing skeleton information, greatly improving the overall accuracy. Moreover, additional tests on KTH and UCF Sports datasets also show the robustness of our approach in the presence of partial body occlusions. Finally, comparisons on UT-Kinect and NTU-RGB+D datasets show that the accuracy of the proposed method is fully comparable to that of works based on 3D skeletons.

I. Introduction

In recent years, vision-related tasks are becoming increasingly important for the scientific community. This is because they can be considered main prerequisites for multimedia content analysis [1]–[3], including event interpretation and behaviour understanding [4], [5]. In [6], for example, a system for automatic detection of crimes (e.g., chain and purse snatching) is reported. To detect variations of the attributes e.g., walking and running) composing these complex actions, the authors propose a Gaussian Mixture Model (GMM) with a large number of mixtures known as Universal Attribute Models (UAMs). In [7], instead, an automated video surveillance system to detect anomalous human behaviours is presented. To obtain reliable results, the authors use a Convolutional AutoEncoder (CAE), which identifies the bi-dimensional structures within the image sequences. Another system designed for monitoring purposes is described in [8], where the authors adopt a Bag-of-Visual-Words (BoVW) model to distinguish different classes of actions. Finally, in [9], [10], are even reported two action recognition methods focused on the evaluation of impaired patients that perform rehabilitative exercises, namely, serious games. In these works, each exercise is treated as a long action, whose evaluation provides the recovery rate of lost skills.