Mode Connectivity in Federated Learning with Data Heterogeneity

Federated learning (FL) allows multiple clients to train a global model while keeping data locally. It has been well recognized that FL suffers from data heterogeneity, leading to drifts in client updates and deteriorating global model performance. However, there is a lack of research on the relationship between client and global models in the parametric space, leaving unclear where client updates end their drifts. To bridge this gap, we consider different model solutions to the client and FL objectives (referred to as client modes and global modes, respectively) and use the mode connectivity of neural networks to measure model performance change along different parametric paths. Specifically, we conduct comprehensive studies on the linear and non-linear connectivity of client modes and global modes to reveal their geometric relationship in the solution space. We find that as data heterogeneity reduces in FL, the connectivity (e.g., performance change) between client and global modes on different paths becomes more similar. This results in more overlapping low-error solutions between client and FL objectives in the solution space. Furthermore, when two global modes are linearly connected, a connectivity barrier emerges, which weakens as heterogeneity decreases. This barrier disappears and becomes independent of data heterogeneity when nonlinear mode connectivity is considered. Our empirical results demonstrate that client and global modes are close to each other in the solution space while they differ in connectivity under varying data heterogeneity.