Two-Stage Adaptive Gradient Quantization in Federated Learning

As the training tasks become more complex, the models become lager with more parameters, and the limited communication resources are still the main bottleneck of federated learning. Some studies have proposed many methods to reduce the communication consumption in federated learning. However, in previous studies, they ignored the impact of different gradient distributions on quantization strategies. Therefore, our study proposes an adaptive gradient compression method(AGCML) which realizes quantization process from two stages, gradient division and gradient quantization. In the first stage, gradient are divided into two parts according to the importance of the gradient. And in the second stage, we select different quantization strategies to quantize the two parts. Our experiments demonstrate that the method we proposed achieve higher model accuracy and compression rate.