GSASG: Global Sparsification With Adaptive Aggregated Stochastic Gradients for Communication-Efficient Federated Learning

This article addresses the challenge of communication efficiency in federated learning by the proposed algorithm called global sparsification with adaptive aggregated stochastic gradients (GSASGs). GSASG leverages the advantages of local sparse communication, global sparsification communication, and adaptive aggregated gradients. More specifically, we devise an efficient global top-
$k^{\prime }$
sparsification operator. By applying this operator to the aggregated gradients obtained from the top-k sparsification, the global model parameter is rarefied to reduce the download transmitted bits from
$O(dMT)$
to
$O(k^{\prime }MT)$
, where d is the dimension of the gradient, M is the number of workers, T is the total number of epochs, and
$k^{\prime } \leq k\lt d$
. Meanwhile, the adaptive aggregated gradient method is adopted to skip meaningless communication and reduce communication rounds. The deep neural network training experiment demonstrates that, compared to the previous algorithms GSASG significantly reduces communication cost without sacrificing the model performance. For instance, when considering the MNIST data set with
$k=1\% d$
and
$k^{\prime }=0.5\% d$
, in terms of communication rounds, GSASG outperforms sparse communication by 91%, adaptive aggregated gradients by 90%, and the combination of sparse communication with adaptive aggregated gradients by 56%. In terms of communication bits, GSASG yields 1% of the communication bits needed with previous algorithms.