TidalMesh: Topology-Driven AllReduce Collective Communication for Mesh Topology

In deep learning workloads, collective communication across multiple nodes is a critical component in determining overall performance. AllReduce (as well as ReduceScatter and AllGather) is a commonly used collective communication for not only training but also inference. The performance of AllReduce depends on the algorithm utilized (or the “logical” topology) as well as the physical topology of the system that interconnects the nodes together. There has been many work on improving AllReduce performance but prior work have often been topologyaware approach where existing AllReduce algorithms were optimized for a given physical topology. In this work, we propose a topology-driven approach where the topology characteristics are exploited to propose a novel AllReduce collective communication algorithm; thus, the logical topology of the algorithm maps well to the physical topology. In particular, as 2D mesh topology is widely used in various scale-out systems, we propose TidalMesh AllReduce algorithm - a novel approach that exploits the inherent characteristics of the physical 2D mesh topology by pushing flows between the endpoint nodes, similar to a tidal wave, to achieve near-optimal performance for AllReduce. We propose how Sparse TidalMesh AllReduce minimizes bandwidth overhead of TidalMesh with no loss in performance. In addition, we demonstrate how collective communication unrolling can be exploited to enable “software pipelining” of collective communication while exploiting the unique opportunity of superimposing different phases of AllReduce. As a result, TidalMesh results in up to $\mathbf{2 4 \%}$ improvement in AllReduce performance across various deep learning models on a 64 -node $8 \times 8$ 2D mesh, compared to the state-of-the-art while maintaining the simplicity of a logical ring algorithm.