Understanding Loss Landscape Symmetry in Federated Learning: Implications for Model Fusion and Optimization

Understanding the optimization landscape in deep neural networks is essential for evaluating model performance, especially in distributed environments like federated learning (FL). Recent studies highlight the importance of valley symmetry in the loss landscape, showing how factors such as dataset heterogeneity, network architecture, and initialization affect convergence. This review delves into these findings within the FL context, stressing the significance of sign consistency in model updates and its impact on model fusion and parameter alignment. Our analysis lays the groundwork for creating more resilient FL systems capable of handling data heterogeneity and asynchronous updates effectively.