Accelerating Federated Learning on Non-IID Data Against Stragglers

In large-scale federated learning systems, it is common to observe straggler effect from those clients with slow speed to delay the overall learning. However, in the standard federated learning frameworks (e.g., FedAvg) on non-iid data distribution among heterogeneous clients, we need to wait for all the clients' updates in each iteration as stopping each iteration in the meantime will change the original data distribution for the learning result. To make up the missing updates from straggler clients and accelerate the federated learning on non-iid data, we are the first to propose a novel method called stopping-and-weight-balancing (SAWB) to stop strategically each iteration and balance the update weights across the non-straggler clients. We prove that this method can keep the original data distribution, and controls weight divergence well with limited accuracy loss from FedAvg and the centralized learning. Finally, using different datasets (MNIST and CIFAR-10) for federated learning experiments, we show that our method can greatly save training time for a large-scale system while preserving the accuracy of the learning result.