Power-effiicent resource allocation in MapReduce clusters

MapReduce has recently evolved in data-intensive parallel computing. It is a programming model for processing large data sets. The implementation of MapReduce typically runs on a large scale of cluster computing systems consisting of thousands of commodity machines. Such cluster computing systems are called MapReduce clusters. The high power consumption of MapReduce clusters has become a major concern since hundreds of MapReduce programs are implemented and thousands of MapReduce jobs are executed in such clusters like Amazon's Elastic MapReduce Clusters every day. Power management becomes one of the most important problems in MapReduce clusters. Furthermore, the availability of MapReduce clusters plays an essential role in the delivery of quality of services (QoS) for customer services. In this paper, we investigate the problem of resource allocation for power management in MapReduce clusters. Specifically, we propose resource allocation approaches to minimizing the mean end-to-end delay of customer jobs or services under the constraints of the energy consumption and the availability of MapReduce clusters and to minimizing the energy consumption of MapReduce clusters under the availability of MapReduce clusters and the mean end-to-end delay of customer jobs or services. Numerical experiments demonstrate that the proposed approaches are applicable and efficient to solve these resource allocation problems for power management in MapReduce clusters.