A Survey of Communication Optimization in Distributed Training

With the increasing scale of deep neural network models, distributed training has become the mainstream approach for accelerating large-scale model training. However, frequent parameter synchronization and intensive communication among large GPU clusters introduce considerable overhead, which has become a critical bottleneck limiting both training efficiency and system scalability. To address this challenge, this paper systematically reviews communication optimization strategies in distributed training, which can be broadly categorized into three main approaches. First, algorithm-level optimizations, which are independent of the underlying network infrastructure, aim to reduce communication volume and the number of communication rounds through techniques such as gradient compression and synchronous updates. Second, system and protocol-level optimizations focus on enhancing transmission efficiency by improving communication protocols, network topologies, and scheduling strategies. Third, routing-level optimizations integrate task characteristics with network structures to mitigate link congestion and reduce communication latency. Furthermore, this paper summarizes the advantages, limitations, and application scenarios of existing mainstream communication optimization techniques and explores future research directions, with the goal of providing valuable insights for the design and optimization of next-generation distributed training systems.