Job Classification for MapReduce Scheduler in Heterogeneous Environment

Hadoop MapReduce is one of the most popular publicly available frameworks for big data processing in the cloud environment. Hadoop provides for the needs of a wide variety of possible users but does not provide a means to optimize the scheduler for individual users. This is what we have attempted to do. The MapReduce scheduling system consists of a single Job Tracker per cluster and multiple Task Trackers. One of the primary responsibilities of the job tracker is to schedule all the user jobs which makes optimizing the scheduling by the Job Tracker an interesting problem. For an improved scheduling framework, we have implemented scheduling which also takes into account the actual resource requirements of the job, as opposed to relying completely on number of free Map and Reduce slots. In this paper, an attempt is made to create a scheduler that can learn and adapt itself to any possible application. The scheduler classifies the tasks to be assigned into two classes, schedulable and non-schedulable. This process weeds out jobs that are unlikely to run on a node using a process that is computationally cheap. The experimentation result detects the job which will overload a particular node and indicate the same to the scheduler. Thus, if it is highly unlikely that a job will successfully run on a particular node, that job will be classified as non-schedulable and not be considered for execution on that node. This will prevent jobs from executing partially, then failing and having to be rescheduled.