An enhanced hadoop heartbeat mechanism for MapReduce task scheduler using dynamic calibration

MapReduce is a popular programming model for processing large-scale datasets in a distributed environment and is a fundamental component of current cloud computing and big data applications. In this paper, a heartbeat mechanism for MapReduce Task Scheduler using Dynamic Calibration (HMTS-DC) is proposed to address the unbalanced node computation capacity problem in a heterogeneous MapReduce environment. HMTS-DC uses two mechanisms to dynamically adapt and balance tasks assigned to each compute node: 1) using heartbeat to dynamically estimate the capacity of the compute nodes, and 2) using data locality of replicated data blocks to reduce data transfer between nodes. With the first mechanism, based on the heartbeats received during the early state of the job, the task scheduler can dynamically estimate the computational capacity of each node. Using the second mechanism, unprocessed Tasks local to each compute node are reassigned and reserved to allow nodes with greater capacities to reserve more local tasks than their weaker counterparts. Experimental results show that HMTS-DC performs better than Hadoop and Dynamic Data Placement Strategy (DDP) in a dynamic environment. Furthermore, an enhanced HMTS-DC (EHMTS-DC) is proposed by incorporating historical data. In contrast to the "slow start" property of HMTS-DC, EHMTS-DC relies on the historical computation capacity of the slave machines. The experimental results show that EHMTS-DC outperforms HMTS-DC in a dynamic environment.