Asynchronous Federated Learning in Decentralized Topology Based on Dynamic Average Consensus

Federated learning (FL) is an emerging technique to tackle the problem of isolated data islands. Vanilla FL usually relies on a centralized topology with a synchronous communication setup, where a parameter server coordinates the distributed users by periodically receiving the model updates from them, performing the aggregation to build a global model and sending back the aggregated model to those users. These restrictions however make FL become vulnerable to the single point failure and not efficient enough given the heterogeneous device capabilities and communication conditions. This paper sheds light on enabling FL in a decentralized topology with an asynchronous communication setup. We first incorporate the asynchronous first-order dynamic average consensus into FL and propose an underlying Async-DFL algorithm. Then, to alleviate the staleness effect in Async-DFL, we introduce a hyper-parameter into Async-DFL and further devise an optimized hyper-mix Async-DFL solution. The experimental result demonstrates the feasibility of these approaches and shows that the hyper-mix solution performs better than the underlying Async-DFL.