ACE-pFL: Accurate, Efficient Personalized Federated Learning With Knowledge Distillation

Personalized Federated Learning (pFL) can collaboratively personalize models for multiple clients without sharing their private data. However, many pFL methods rely on server-side model parameters aggregation, which requires all models to have the same structure and size. One promising approach is leveraging knowledge distillation (KD) to transfer knowledge between models by exchanging soft predictions rather than model parameters, thus training heterogeneous models. Nevertheless, existing KD-based pFL solutions suffer from accuracy loss due to inadequate knowledge extraction as well as huge computing and communication overheads. In this paper, we present an accurate and efficient KD-based pFL framework, called ACE-pFL. Specifically, we first propose a privacy-preserving client clustering to reduce the impact of non-independent and identically distributed (non-IID) data on model accuracy and convergence, grouping clients with similar data distributions into the same cluster. Since the distillation temperature of traditional KD is fixed, which does not consider the dynamic model training process, we design a dynamic distillation temperature adjustment to accommodate this process, where clients incrementally increase the distillation temperature as training proceeds to facilitate model generalization to new data. Finally, we employ the triple distillation strategy to provide diverse and abundant knowledge, including explicit global knowledge, implicit local knowledge, and implicit global knowledge. Experiments on multiple datasets and tasks show that compared with existing schemes, ACE-pFL can significantly improve the test accuracy by 17.18%, reduce the training time by 57% and the communication overhead by $59.12\times $ on average.