Far-Sighted Multi-Stage Awasre Coflow Scheduling

In data center networks (DCN), large scale flows produced by parallel computing frameworks form many coflows semantically. Most inter-coflow schedulers only focus on the remaining data of coflows and attempt to mimic Shortest Job First (SJF). However, a coflow may consist of multiple stages. In this paper, we consider the Multi-stage Inter-Coflow Scheduling problem and try to give an efficient online scheduling scheme. We first explore a short-sighted algorithm with the greedy strategy. This gives us an insight into utilizing the network resources. Based on that, we propose a far-sighted heuristic, which schedules sub-coflows to occupy network bandwidth in turn. Through simulations in various network environments, we show that, compared to a state-of-the-art scheduler - Varys, a multi-stage aware scheduler can reduce the coflow completion time by up to \pmb4.81× even though it is short-sighted. Moreover, the far-sighted scheduler can improve the performance by nearly \pmb7.95 × reduction.