Parallel Job Scheduling through Evolutionary Based Cognitive Strategies

Traditional schedulers for high performance computing (HPC) systems are, nowadays, powerful but hard to configure, with a large number of parameters. They are not very flexible and they ignore two things: many users don't know or specify the resources needed by their jobs, and the same system performance is not perceived equally by every user. The work presented here is focused on the task of improving scheduling by addressing these problems through a system based on three key components: genetic algorithms, user behavior models and user satisfaction models. Thus, a genetic algorithm tries to find, with no human intervention, the best order for the execution of jobs using an automatically obtained behavior model for each user that predicts the real amount of resources needed, and a fitness function that takes into account the concept of user satisfaction (extracted from a user satisfaction model) in addition to classical parameters such as makespan or waiting times. The complete scheduling system is described as well as its integration in the Sun Grid Engine (SGE). Some experiments are carried out to compare its behavior to that of the SGE, including the effects of using different parameters in the fitness function in order to consider different needs or policies of the HPC center. Some comments are provided on how user satisfaction models affect scheduling.