Federated Adaptive Pseudo-Labeling Selection for Semi-Supervised Air Writing Recognition Systems

The rapid advancement of virtual and augmented reality technologies has catalyzed a new paradigm in consumer electronics: air writing, a form of touchless human-computer interaction with vast potential. Implementing air writing recognition systems, however, faces challenges such as label scarcity and privacy concerns. Addressing these, we propose the Federated Adaptive Pseudo-labeling Selection (FedAPS) framework, a federated semi-supervised learning approach, enhancing decision-making in consumer electronics using multi-modal data. FedAPS innovatively utilizes limited labeled data alongside extensive unlabeled data, maintaining user privacy. We employ a multi-modal data augmentation process for air writing recognition by designing an adaptive pseudo-labeling strategy. This enables clients to select the most appropriate model for pseudo-labeling based on historical local and global models and dynamic word score recommendations. We introduce a historical local-global consistency regularization to maximize knowledge extraction from unselected models when similar predictions occur. Our comprehensive evaluation on a real-world multi-modal air writing dataset shows FedAPS’s effectiveness, outperforming advanced federated semi-supervised baselines and achieving performance comparable to fully labeled federated supervised learning. This highlights its potential in enhancing data-driven decision-making for next-generation multi-modal input consumer electronics.