Fed-PID: An Adaptive Learning Rate Scheduler for Federated Learning with PID Controllers

The adaptation of learning rate can significantly enhance the performance of deep neural networks, especially in intricate environment of federated learning (FL) with diverse client settings. Therefore, the hyperparameter optimization is a critical component in FL. Distributed iterative optimization algorithms for FL encounter issues such as slow convergence due to inter-node coupling, as well as diminished inference accuracy. Furthermore, the reliance on laborious empirical learning rate adjustment results in weak adaptability to changing environments and slow responsiveness to dynamic conditions. In this work, we propose Fed-PID, a novel PID controller based adaptive learning rate scheduler for FL. Fed-PID employs distributed feedback control and PID technique to re-express distributed iterative optimization algorithms as distributed PID controllers. By equivalently transforming multiple coupled control systems into an extended dimensional error system, it converts the federated aggregation convergence problem into a consensus control problem. We design Server Side Parameter Scheduler (SSPS) to automatically adjusts the P, I, and D parameters for all clients by minimizing the extended dimensional error system from a global perspective and design Client Side Parameter Schedulers (CSPS) to automatically tune hyperparameters during local training from a micro perspective. Extensive experimental results on three public datasets and our self-built cobot radar localization dataset demonstrate that Fed-PID consistently outperforms FedAvg and other baseline methods. In particular, Fed-PID outperforms its competitors by 3.6% accuracy on CIFAR-10 and 5.85% on Cobot dataset under the Non-IID setting.