UnisonFlow: A Software-Defined Coordination Mechanism for Message-Passing Communication and Computation

Message passing interface (MPI) communication performance is becoming one of the key factors heavily affecting the total performance of data-intensive applications running on computer clusters. Our software-defined networking (SDN)-enhanced MPI improves the performance of communication over interconnects by integrating flexible and dynamic network controllability of SDN into MPI. We have demonstrated that the acceleration of individual MPI communication primitives is feasible through our past work on the SDN-enhanced MPI. However, real-world MPI applications have not benefited from such accelerated communication primitives through our research achievements to date, because each of the distinct network control algorithms designed for various MPI communication primitives cannot be activated and coordinated with the execution of the MPI application. Therefore, this paper proposes UnisonFlow, a software-defined coordination mechanism for the SDN-enhanced MPI that performs network control in synchronization with the execution of applications. An experiment conducted on a real-computer cluster verifies that the interconnect control can be successfully performed in synchronization with the execution of the application. Furthermore, the synchronization is performed with a low overhead and its performance penalty is practically negligible.