Object-Augmented Skeleton-Based Action Recognition

Human Skeleton-based Action Recognition (SAR) methods have made great advances in different applications in recent years. However, most existing SAR models mainly focus on the joint/limb pose estimation. They ignore the skeleton-object interaction, thereby resulting in underutilization of information available. For instance, drinking and eating actions may exhibit similar skeleton movements but different object interactions. Hence relying on joint/limb pose estimation alone may lead to incorrect action predictions; but leveraging skeleton-object interaction will help to discriminate such similar actions. In view of this, we propose a new effective method called Object-Augmented Skeleton-based Action Recognition (OA-SAR) to integrate skeleton-object interactions into action recognition. In specific, OA-SAR extracts the object and joint/limb heatmaps, and then integrates this information for subsequent action recognition. We evaluate the OA-SAR method on two benchmarks, NTU-RGB+D-60, and Drive&Act. Experimental results show that OA-SAR can achieve strong performance on both action datasets.