A Cuboid CNN Model With an Attention Mechanism for Skeleton-Based Action Recognition

The introduction of depth sensors such as Microsoft Kinect have driven research in human action recognition. Human skeletal data collected from depth sensors convey a significant amount of information for action recognition. While there has been considerable progress in action recognition, most existing skeleton-based approaches neglect the fact that not all human body parts move during many actions, and they fail to consider the ordinal positions of body joints. Here, and motivated by the fact that an action's category is determined by local joint movements, we propose a cuboid model for skeleton-based action recognition. Specifically, a cuboid arranging strategy is developed to organize the pairwise displacements between all body joints to obtain a cuboid action representation. Such a representation is well structured and allows deep CNN models to focus analyses on actions. Moreover, an attention mechanism is exploited in the deep model, such that the most relevant features are extracted. Extensive experiments on our new Yunnan University-Chinese Academy of Sciences-Multimodal Human Action Dataset (CAS-YNU MHAD), the NTU RGB+D dataset, the UTD-MHAD dataset, and the UTKinect-Action3D dataset demonstrate the effectiveness of our method compared to the current state-of-the-art.

I. Introduction

Human action recognition [1]–[5] is an active yet challenging research area that has been explored in many applications including healthcare, smart surveillance, and security. RGB sensors and depth sensors (e.g., Microsoft Kinect sensors) have been used to improve human action recognition performance by exploiting rich, captured information such as depth and 3D location. Compared to RGB data, depth data can adapt to changes in lighting conditions through the use of infrared radiation. Xiao et al. [6] and Ji et al. [7] proposed an effective method to recognize human actions from depth map sequences. However, due to redundancy in depth maps, this huge amount of data increases computational complexity making them impractical for real-world use.