Implementing efficient and scalable flow control schemes in MPI over InfiniBand

Summary form only given. We present a detailed study of how to design efficient and scalable flow control mechanisms in MPI over the InfiniBand architecture. Two of the central issues inflow control are performance and scalability in terms of buffer usage. We propose three different flow control schemes (hardware-based, user-level static and user-level dynamic) and describe their respective design issues. We have implemented all three schemes in our MPI implementation over InfiniBand and conducted performance evaluation using both microbenchmarks and the NAS parallel benchmarks. Our performance analysis shows that in our testbed, most NAS applications only require a very small number of preposted buffers for every connection to achieve good performance. We also show that the user-level dynamic scheme can achieve both performance and buffer efficiency by adapting itself according to the application communication pattern. These results have significant impact in designing large-scale clusters (in the order of 1,000 to 10,000 nodes) with InfiniBand.