Efficient Federated Learning for Heterogeneous Data: A Client Selection and Adaptive Knowledge Distillation Approach

Federated Learning (FL) is a distributed machine learning paradigm where multiple clients train a shared global model collaboratively. FL was developed in response to growing concerns regarding data ethics and privacy. However, the communication overhead inherent to FL can be a significant bottleneck. Client devices containing heterogeneous data distributions constitute another key difficulty to the training of the shared model until its convergence. This investigation introduces AdaptiveKDS, a novel approach that integrates client selection and knowledge distillation to mitigate these challenges. By strategically selecting a subset of clients in each training round, we accelerate the convergence speed of the shared model. Our adaptive local knowledge distillation mechanism customizes the knowledge transfer process to each client’s local data. This approach effectively tackles the challenge posed by heterogeneous local data distributions. Our empirical evaluation, conducted on the MNIST and the German Traffic Sign Benchmark datasets, demonstrates that AdaptiveKDS consistently outperforms existing FL methods regarding the global convergence speed, overall accuracy and performance of the global model on individual client datasets.