Personalized Federated Learning based on Knowledge Distillation using Teacher Assistants

Federated Learning (FL) is a privacy-preserving distributed machine learning paradigm. However, the heterogeneity of data among clients leads to the shared global model obtained after training being unable to adapt to the distribution of each client's dataset, resulting in a decrease in model performance. To address this issue, this paper proposes a personalized federated learning method based on teacher assistant knowledge distillation-pFedCo- TA. This method clusters clients based on the similarity of distribution information of the datasets uploaded by different clients before training. By clustering clients, those with similar data distributions are grouped into a class, an assistant is appointed, and then knowledge distillation is performed between teacher-assistant and assistant-student to compress the model based on knowledge extraction, reducing communication costs, and improving the accuracy of non-independent and identically distributed (non-IID) data through the assistant's intermediate layer. This method effectively alleviates these bottlenecks and improves the performance gap of non-IID data models. Finally, we conduct simulation experiments on different data heterogeneous scenarios on two datasets. Compared to single-model algorithms, pFedCo- TA achieves an average accuracy improvement of 27.1 % and 12.49% over FedAvg and FedProx, respectively; compared to typical personalized algorithms, pFedCo-TA's accuracy is on average 6.7% higher than FedMD.