By Topic

Data-Intensive Workload Consolidation for the Hadoop Distributed File System

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

5 Author(s)
Moraveji, R. ; Sch. of Inf. Technol., Univ. of Sydney & Nat. ICT Australia (NICTA), Sydney, NSW, Australia ; Taheri, J. ; Farahabady, M.R.H. ; Rizvandi, N.B.
more authors

Workload consolidation, sharing physical resources among multiple workloads, is a promising technique to save cost and energy in cluster computing systems. This paper highlights a number of challenges associated with workload consolidation for Hadoop; as one of the current state-of-the-art data-intensive cluster computing systems. Through a systematic step-by-step procedure, we investigate challenges for efficient server consolidation in Hadoop environments. To this end, we first investigate the inter-relationship between last level cache (LLC) contention and throughput degradation for consolidated workloads on a single physical server employing Hadoop distributed file system (HDFS). We then investigate the general case of consolidation on multiple physical servers so that their throughput never falls below a desired/predefined utilization level. We use our empirical results to model consolidation as a classic two-dimensional bin packing problem and then design a computationally efficient greedy algorithm to achieve minimum throughput degradation on multiple servers. Results are very promising and show that our greedy approach is able to achieve near optimal solutions in all experimented cases.

Published in:

Grid Computing (GRID), 2012 ACM/IEEE 13th International Conference on

Date of Conference:

20-23 Sept. 2012