Challenges and Insights in Cross-dataset Skeleton-based Action Recognition

Skeleton-based action recognition methods have achieved impressive performance in recent years. However, when applied to real-world scenarios, these models often suffer from performance degradation due to external factors such as variations in hardware, environment, viewpoint, and human morphology. In this study, we investigate the impact of different domains on model performance under cross-dataset settings. Specifically, we construct two skeleton-based action recognition benchmarks, NTU RGB+D 120-16 (NTU120-16) and ETRI-Activity3D-16 (EA3D-16), by selecting identical action categories from the NTU120 and EA3D datasets. We conduct extensive experiments using these datasets as source and target domains interchangeably. Our results reveal a significant performance drop when transferring models across datasets. When trained on NTU120-16 and tested on its own test set, the model achieves an accuracy of 88.55%. However, when applied to EA3D-16, the accuracy drops sharply to 55.53%, resulting in a 33.02% decrease. Similarly, when trained on EA3D-16 and tested on NTU120-16, the accuracy declines by 47.09%. To mitigate this cross-domain performance gap, we introduce a spatial-temporal cross-attention mechanism that enhances the interaction between spatial and temporal features. By leveraging a bidirectional attention mechanism, our method dynamically aligns spatial and temporal representations across domains, enabling the model to focus more effectively on shared action characteristics. Experimental results demonstrate that our method consistently improves cross-dataset performance, highlighting its effectiveness in addressing the challenges of cross-domain skeleton-based action recognition.