NUMA Aware I/O in Virtualized Systems

In a Non Uniform Memory Access (NUMA) system, I/O to a local device is more efficient than I/O to a remote device. A device connected to the same socket having the CPU and memory offers a closer proximity for I/O operations. Modern microprocessors also support on-chip I/O interconnects with which the processor may drive I/O to a local device from the processor cache, without the need to interact with main memory. Modern systems are also highly virtualized. In NUMA systems, scheduling of Virtual Machines (VMs) is very complex because multiple VMs are competing for CPU, memory and devices spread across NUMA nodes. In this paper, we study how to schedule VMs for better I/O on a NUMA system. We propose the design of a NUMA aware I/O hypervisor scheduler, which aligns VMs, hypervisor threads on NUMA boundaries, while extracting the most benefit from local device I/O. Wedemonstrate a benefit of more than 25% in throughput and packet rate, a benefit of more than 10% in CPU utilization, and a benefit of higher than 5% in VM consolidation ratio.