Leveraging Interconnect QoS Capabilities for Congestion-Aware MPI Communication

Resource sharing between job allocations on production supercomputers often leads to traffic interference between MPI applications. Network congestion, a side effect of such resource sharing, results in substantial MPI latency degradation, making application performance unpredictable. We consider the traffic isolation capabilities of modern RDMA interconnects and propose a generic Priority Assignment algorithm that associates a priority with each network Send operation using information about the network latency in the past. Our implementation of Priority Assignment algorithm in UCX framework shows up to the 22 x GPCNeT benchmark improvement on 64 node cluster with InfiniBand EDR and OpenMPI in comparison to the reference UCX implementation. Packet-level simulation shows that proposed Priority Assignment algorithm helps to mitigate the effects of network congestion scaling up to the 512 nodes.