Rate-Distortion Optimization for Adaptive Gradient Quantization in Federated Learning

Federated learning (FL) is an emerging machine learning setting designed to preserve privacy. However, constantly updating model parameters on uplink channels results in huge communication overload, which is a major challenge for FL. In this paper, we consider an adaptive gradient quantization approach based on rate-distortion optimization in FL, which consists of a non-stationary random walk model on the true global optimal model parameters. Unlike traditional quantization methods, our goal is to minimize the total communication costs when the global server reconstructs model parameters under distortion constraints. Furthermore, when considering the iterative process, we utilize the Kalman filter to reduce computational complexity. And in each iteration, a generalized water-filling algorithm is used to calculate the optimal quantization levels for each local client. Numerical results show that the proposed method outperforms conventional quantization methods in terms of reducing communication costs.