FedFA: Federated Learning With Feature Anchors to Align Features and Classifiers for Heterogeneous Data

Federated learning allows multiple clients to collaboratively train a model without exchanging their data, thus preserving data privacy. Unfortunately, it suffers significant performance degradation due to heterogeneous data at clients. Common solutions involve designing an auxiliary loss to regularize weight divergence or feature inconsistency during local training. However, we discover that these approaches fall short of the expected performance because they ignore the existence of a
vicious cycle
between feature inconsistency and classifier divergence across clients. This
vicious cycle
causes client models to be updated in inconsistent feature spaces with more diverged classifiers. To break the
vicious cycle
, we propose a novel framework named
Fed
erated learning with

F
eature

A
nchors
(FedFA). FedFA utilizes feature anchors to align features and calibrate classifiers across clients simultaneously. This enables client models to be updated in a shared feature space with consistent classifiers during local training. Theoretically, we analyze the non-convex convergence rate of FedFA. We also demonstrate that the integration of feature alignment and classifier calibration in FedFA brings a
virtuous cycle
between feature and classifier updates, which breaks the
vicious cycle
existing in current approaches. Extensive experiments show that FedFA significantly outperforms existing approaches on various classification datasets under label distribution skew and feature distribution skew.