Client-Unbiased Skeletal Action Recognizer in Federated Learning

Edge sensor devices generate vast amounts of user data, but centralized processing poses privacy risks. Federated Learning addresses this by decentralizing training. However, applying Federated Learning directly to skeleton videos fails to preserve motion dynamics and suffers from client heterogeneity bias. To address these limitations, we propose CSAR—a Client-Unbiased Skeletal Action Recognizer for Federated Learning—which tackles two core challenges: motion dynamics preservation and classifier bias mitigation. Specifically, CSAR employs a Model Calibration Loss during client training to align client-server representations and reduce drift. On the server, it generates class-balanced spatiotemporal federated features through Prototypical Gaussian Sampling, subsequently refined via a Motion-aware Differential Loss to capture kinematic properties. These features enable retraining of a globally debiased recognizer that achieves accuracy comparable to real-data-trained models. Further stabilization is achieved through Knowledge Matching, which enhances global understanding. Experiments under natural and label heterogeneity confirm that CSAR outperforms state-of-the-art methods.