Stragglers Are Not Disasters: A Hybrid Federated Learning Framework with Delayed Gradients

Federated learning (FL) is a new machine learning framework that trains a joint model across a large number of decentralized computing devices. Existing methods, e.g., Federated Averaging (FedAvg), are able to provide an optimization guarantee by synchronously training the joint model, but usually suffer from stragglers, i.e., IoT devices with low computing power or communication bandwidth, especially on optimization problems with non-i.i.d distributed training data. To mitigate the influence of stragglers, this paper presents a novel FL framework, namely Hybrid Federated Learning (HFL), which consists of two major components: synchronous kernel and asynchronous updater that correspondingly addresses the normal devices and stragglers. Unlike traditional synchronous FL frameworks, HFL introduces the asynchronous updater which actively pulls unsyn-chronized and delayed local updates from stragglers to improve the efficiency and effectiveness of FL. Particularly, we propose a novel algorithm, delayed federated-SGD (DF-SGD), to merge the delayed local updates into the joint model. The results of comprehensive experiments under the HFL framework show that DF-SGD outperforms current FL methods.