Profile driven scheduling for a heterogeneous server cluster

Clusters of commodity servers are increasingly the platform of choice for running compute-intensive jobs in a variety of industries. Computations such as wind-tunnel simulations, gene and protein analysis, drug discovery, and CGA rendering are run on commodity computers with very successful results. At the same time most cluster environments employ a highly heterogeneous set of machines due to the natural cycle of upgrades and purchases followed by most organizations. In such environments it is likely that some applications may be better suited for one type of machine over another and that the placement of jobs on machines can have a large impact on throughput and job completion time. This paper argues that using profile information to guide scheduling decisions can yield substantial performance improvements over a simple FCFS scheduling policy. In particular we look at a real-world job mix of genomic analysis applications and examine a number of scheduling algorithms that take profile information into account. Our results indicate that the benefits of profile-driven scheduling vary significantly based on the degree of heterogeneity available in the cluster and the variance in execution times of different job types across cluster machines. Additionally, we have discovered that affinity effects can also play an important role in improving performance, but plain affinity scheduling is not sufficient in achieving those benefits.