Decentralized Machine Learning Training: A Survey on Synchronization, Consolidation, and Topologies

Federated Learning (FL) has emerged as a promising methodology for collaboratively training machine learning models on decentralized devices. Notwithstanding, the effective synchronization and consolidation of model updates originating from diverse devices, in conjunction with the appropriate configuration of network topologies, persist as crucial obstacles. This paper provides a comprehensive analysis of the current techniques and methodologies utilized in the synchronization, consolidation, and network topologies of Federated Learning. The present study explores diverse synchronization strategies utilized for the purpose of coordinating model updates from geographically distributed cross-silo edge nodes. The study takes into account several factors, including communication efficiency and privacy preservation. This study delves into the intricacies of model consolidation techniques, such as weighted and personalized aggregation methods, to evaluate their efficacy in consolidation of local model updates into a global model, while taking into consideration statistical heterogeneity and resource constraints. In addition, an examination is conducted on the importance of network topologies in Federated Learning (FL), taking into account their influence on communication efficacy, confidentiality, expandability, resilience, and resource allocation. The survey assesses and contrasts the efficacies and constraints of extant methodologies, discerns deficiencies in present investigations, and provides insights for future progressions. The objective of this survey is to provide a thorough examination of FL synchronization, consolidation, and network topologies, with the intention of offering a valuable reference for individuals engaged in Federated Learning, including researchers, practitioners, and stakeholders. This survey aims to support the advancement of more effective and resilient FL systems.