CODE: Incorporating Correlation and Dependency for Task Scheduling in Data Center

The popularity of data-parallel applications and big data platforms has led to a variety of communication patterns in data center, which brings more complexity to task scheduling. Existing works try to minimize the overall completion time (i.e. makespan) of tasks but fail to take a full consideration of the relationship among tasks. Focusing on this, we propose a novel task scheduling scheme, CODE, which incorporates both the correlation and dependency among tasks for scheduling. We formulate the problem as a variant of Flexible Job Scheduling Problem (FJSP) to minimize the makespan. Firstly, we propose a novel network abstraction to describe the tasks as well as their relationship. Then the problem is formulated into an extensive FJSP and we solve it with a Tabu Search Algorithm. The evaluation has demonstrated the effectiveness of our method. Specifically, there is a reduction of over 16% in makespan with CODE compared with baseline scheduling scheme.