Multi-Stream Longtail Graph Convolutional Network for Human Action Recognition

Human action recognition (HAR) has benefited significantly from the application of graph convolutional networks (GCNs), which model the topological relationships between joints. In the consumer electronics field, this technology has brought revolutionary innovations to smart home devices, wearable devices, and game interaction systems. However, existing GCN-based methods often process spatial-temporal correlations simultaneously, neglecting subtle joint relationships across consecutive frames. To address this limitation, this paper proposes a multi-stream longtail graph convolutional network (MSL-GCN), which effectively leverages spatial-temporal information between joints and handles tasks involving multi-stream fusion well. The proposed MSL-GCN incorporates a multimodal data modelling strategy based on the fusion of two-dimensional (2D) RGB and three-dimensional (3D) skeleton data to extract skeleton features comprehensively. Additionally, it introduces a spatial-temporal decoupling of the GCN branch to capture joint correlations across spatial and temporal dimensions efficiently. To enhance generalizability and robustness, a multi-stream longtail classification strategy is developed, enabling the model to adapt to imbalanced data distributions and complex real-world scenarios. Extensive experiments on the UAV-Human, NTU-RGB+D-60 and NTU-RGB+D-120 datasets show that MSL-GCN achieves the SOTA performance that outperforms existing representative methods, with an improvement of 1.51%, 3.58% and 0.68% over the suboptimal algorithms, respectively. Meanwhile, the experimental results show that the multi-stream longtail classification strategy can balance the recognition ability of low-frequency classification and improve the robustness and practicality of the classification model.