FedCos: A Scene-Adaptive Enhancement for Federated Learning

Federated learning (FL) training global machine learning models over distributed edge devices has attracted sustained attentions. However, the heterogeneity of client data severely degrades the performance of FL compared with that in centralized training. On the one hand, it slows down or even stalls global updates, leading to inefficient communication. On the other hand, it enlarges the distances between local models, resulting in an aggregated global model with poor performance. Fortunately, these shortcomings can be mitigated by reducing the angle between the directions in which a local model move. Based on this observation, we propose FedCos, which reduces the directional inconsistency of local models by introducing a cosine-similarity penalty. It promotes local model iterations toward an auxiliary global direction. Moreover, our approach is auto-adapted to various non-identically and independently distributed (IID) settings without an elaborate selection of hyperparameters. Experimental results on both vision and language tasks with a variety of models (including CNN, ResNet, LSTM, etc.) show that FedCos outperforms the well-known baselines and can enhance them under a variety of FL scenes, including varying degrees of data heterogeneity, different number of participants, and cross-silo and cross-device settings. Besides, FedCos improves the communication efficiency by 2–5 times. With the help of FedCos, multiple FL methods require significantly fewer communication rounds than before to obtain a comparable model.