Aggregation-Assisted Proxyless Distillation: A Novel Approach for Handling System Heterogeneity in Federated Learning

System heterogeneity in Federated Learning (FL) is commonly dealt with knowledge distillation by combining the clients’ knowledge via distillation into a global model. However, such knowledge transfer to the global model is often limited by distillation efficiency and unavailability of the client data. Most of the existing approaches require proxy data on the server side for distillation, which often becomes a bottleneck. To circumvent these limitations, we propose a novel FL framework, FedAgPD (Aggregation-Assisted Proxyless Distillation for Heterogeneous Federated Learning) that comprises of deep mutual learning (DML) at client end, and global aggregation followed by noise engineered data-free distillation at the server end. DML enables server side global aggregation which otherwise is infeasible due to different client model architectures. The aggregation results in knowledge integration which is further boosted by the subsequent distillation. We further introduce the idea of selective mutual learning where only those clients perform DML that are not limited by computational capacity. This reduces the overall computational burden without any compromise in the performance. We conduct rigorous experiments on various publicly available datasets and observe a remarkable improvement in the performance over the existing heterogeneous FL methods. For example, for CIFAR100 dataset, FedAgPD shows almost two times better performance as compared to the best baseline. Moreover, we compared FedAgPD with recent homogeneous methods and observed a competitive performance. The results provide evidence for the utility and effectiveness of our approach and open up a new direction for heterogeneous FL. Code for FedAgPD is available at https://github.com/nirbhay-design/FedAgPD