Power Aware NUMA Scheduler in VMware's ESXi Hypervisor

Virtualized platforms have emerged as the top solution for cloud computing, especially in today's power-constrained data centers. Virtualization helps save power and energy by allowing physical machines to be replaced by virtual machines (VMs) and then consolidated onto a smaller number of physical hosts. The number of physical hosts that are powered on can even be dynamically varied, as with VMware's Distributed Power Management (DPM) feature. At a lower level, it remains valuable to manage power usage within each individual host, and typical systems, including VMware's ESXi hypervisor, do so by adjusting each processor's P-states (frequency and voltage states) and Cstates (idle states) according to the demands of the current workload. With current NUMA systems, however, there is an intermediate level of power management possible that has gone largely unexplored. In this paper we propose to optimize the placement of virtual machines on NUMA enabled systems, such that the overall energy consumption of the virtualized system is reduced with minimal impact on VM performance. Our heuristics exploit a relatively new CPU hardware feature, called independent package C-states. To the best of our knowledge, this paper presents the first work on making a NUMA scheduler power-aware by exploiting independent package C-states. We implemented a simple heuristic in ESXi and observed power savings of up to 26% and energy efficiency improvements of up to 30% using four realistic workloads and two micro-benchmarks.