Optimizing the Cost-Performance Tradeoff for Coflows Across Geo-Distributed Datacenters

For data analytics jobs running across geographically distributed datacenters, coflows have to go through the inter-datacenter network over relatively low bandwidth and high cost links. In this case, optimizing cost-performance tradeoffs for such coflows becomes crucial. Ideally, decreasing the coflow completion time (CCT) can significantly improve the network performance, meanwhile, reducing the transmission cost introduced by these coflows is another fundamental goal for datacenter operators. Unfortunately, minimizing both the CCT and the transmission cost are conflicting objectives that cannot be achieved concurrently. Prior methods have significant limitations when exploring such tradeoffs, because they either merely decrease the average CCT or reduce the transmission cost independently. In this paper, we focus on a cost-performance tradeoff problem for coflows running across the inter-datacenter network. Specifically, we formulate an optimization problem, so as to minimize a combination of both the average CCT and the average transmission cost. This problem is inherently hard to solve due to the unknown information of future coflows. Therefore, we present Lever, an online coflow-aware optimization framework, to balance these two conflicting objectives. Without any prior knowledge of future coflows, Lever has been proved to have a non-trivial competitive ratio in solving this cost-performance tradeoff problem. Results from large-scale simulations demonstrate that Lever can significantly reduce the average transmission cost, and at the same time, speed up the completion of these coflows, compared with the state-of-the-art solutions.