Communication Efficient Federated Learning Based on Combination of Structural Sparsification and Hybrid Quantization Sensing

Federated learning obtains the global model through the cooperative training of each participating device while protecting data privacy. However, the huge communication cost caused by each client sending complete model update becomes an important problem for its wide application. Model sparsification and quantization are practical solutions to reduce single-round uplink communication, respectively. However, when the compression ratio increases, (1) the current model sparsification methods based on fine granularity by uploading significant gradients are difficult to effectively select important parameters by using thresholds; (2) the unified bit-width quantization of parameters of layers with different importance in the model will lead to greater information loss. In this paper, we propose a joint compression framework based on functional structure sparsity and hybrid quantization sensing to compress the communication cost to more than two orders of magnitude. First, we proposed to take filters as sparse granularity through the visual analysis of parameter updating rules in the training process, and searched for filters with strong representational ability among filters with different representational abilities in the convolution layer. Second, we propose an adaptive layer-sensing quantization method according to the parameter distribution of each layer, which assigns different bit widths to different layers in the model after structural sparsification. Finally, we design a model aggregation and update scheme based on the above joint compression framework, which reduces the error caused by compression through model reconstruction and parameter reuse. Experiments show that our proposed framework compress single communication to more than two orders of magnitude in different tasks while ensuring the convergence speed and final model performance.