Multi-Path Transmission for Parameter Synchronization in Distributed Deep Learning

The increasing scale of deep learning models requires widespread adoption of distributed training using the Ring-AllReduce (RAR) architecture. However, existing multi-path transmission mechanisms are infeasible to satisfy the high bandwidth requirements of RAR parameter synchronization, which exhibits unique traffic characteristic that the parameter synchronization latency of each batch depends on the slowest communication delay between task nodes. To address this issue, this paper proposes RingMP, which selects multiple paths for parameter transmission by comprehensively considering the characteristics of RAR parameter synchronization traffic and network conditions to reduce network congestion. Experimental results show that compared to existing approaches, RingMP achieves higher network throughput, lower packet loss rate, and lower bandwidth wastage for RAR parameter synchronization.