A Scheduling Algorithm Based on Task Complexity Estimating for Many-Task Computing

There is a very important class of applications which is named Many-Task Computing (MTC). For a lot of MTC applications, a large number of independent tasks which differ significantly on task complexities will be generated. This brings a great challenge for grids to achieve a high performance for such MTC applications. In this paper, we describe the TCE algorithm, a scheduling algorithm based on Task Complexity Estimating which reduces the overhead by applying task bundling. We also present a task complexity model for task complexity estimating in order that after task bundling loads among computing nodes can be well balanced. The TCE algorithm greatly exceeded the other scheduling algorithms involved in performance evaluation on speedup and efficiency, and it achieved a performance close to that in the ideal condition. It is demonstrated that by applying the TCE algorithm the overhead cost can be reduced significantly and that load balance can be well guaranteed, so that grids can achieve a high performance for MTC applications.