Learning Self-Correlation in Space and Time as Motion Representation for Action Recognition

Human skeleton, as a compact representation of human action, has received increasing attention in recent years due to the easy accessibility of the human skeleton data. Many previous works capture local physical dependencies among joints, which may miss implicit joint correlations. In this light, we propose a novel motion representation scheme to infer action categories, which includes self-correlation transformation, feature extraction and feature fusion modules. Our network takes a sequence of 2D skeletal heat map as the input, and captures the correlation across frames and adjacent skeleton key points to reason key motion features. Extensive experiments have been conducted on 3 mainstream skeleton-based action recognition datasets, in which our network exhibits superior performances than other state-of-the-art methods.