Client Availability Aware Privacy-Preserving Federated Learning for Language Models

Language Models (LMs) have significantly advanced various natural language processing applications but pose challenges related to data privacy and the need for massive computational resources. Federated Learning (FL) decentralizes the training across multiple devices, enhancing data privacy. Yet, real-world FL faces complexities stemming from inconsistent client availability, especially in devices affected by factors like usage patterns and timezone correlations. Recent frameworks like CA-Fed have addressed client availability discrepancies, but they are less effective when applied to the nuanced domain of LMs. In this context, we introduce Fed-Cab, a novel Client Availability Aware Privacy-preserving Federated Learning framework tailored for LMs. Leveraging finite-state Markov chains, Fed-Cab simulates real-world client availability. Our framework introduces a novel dynamic weighting mechanism that understands and capitalizes on client availability, placing emphasis on clients that aid optimal convergence. By managing client contributions and introducing an innovative communication design, Fed-Cab efficiently captures client availability, determines weights, and dynamically selects contributors and updates models. Our results indicate that Fed-Cab not only maintains high accuracy but also excels in training speed and consistency over time.