Load Balancing for LLM Traffic via Flow Block

Network load balancing is important in datacenter network for LLM training with low entropy traffic. Through in-depth experimental investigations, we find that traditional flowlet-based switch rerouting schemes face substantial challenges in achieving an optimal balance between flowlet timeout settings and packet reordering issues under LLM traffic. Thus, we present CoLB, an application layer and network layer codesign load balancing scheme. CoLB utilizes size-based flow block instead of a timeout value-based flowlet as the rerouting unit for LLM traffic, where the size of the flow block is adaptive to the collective communication patterns generated by the application workload layer. Besides, CoLB is aware of the link utilization affected by PFC mechanism in the underlying RDMA network and guarantees a more accurate perception of link congestion conditions. We evaluate CoLB with the NS-3 simulator under the collective communication traffic and realistic workloads of LLMs. The simulation results show that CoLB is superior to previous switch-based load balancing schemes. For example, CoLB achieves up to 79.88% lower 99th Flow Completion Time (FCT) slowdown than LetFLow and up to 38.83% than Conweave.