Xen and the Art of Cluster Scheduling

In shared use clusters, scheduling systems must schedule both serial and parallel jobs in a fair manner, while at the same time optimizing overall cluster efficiency. Since serial and parallel jobs conflict considerably, scheduling both types of jobs concurrently is a difficult task. Two major strategies are in common use: partitioning the cluster (thus avoiding the problem) and reservation combined with backfilling. Both have major drawbacks in overall performance, ease of use and fairness depending on the particular configuration, which can lead to heated debates between the users of the different types of jobs. In this paper, we introduce an approach to create dynamic virtual cluster partitions using para-virtualization techniques, to deal with the conflicts between parallel and serial jobs. The system dynamically adjusts to different types of job loads and offers easy and transparent use and configuration to both users and administrators, without resorting to complicated runtime prediction or backfilling algorithms. A proof-of-concept implementation based on the Sun Grid Engine scheduling system and Xen is presented.