A Case for Software-Based Adaptive Routing in NUMA Systems

Memory placement in NUMA systems has a significant impact on overall performance; however, most prior work has not considered the performance impact of NUMA interconnect as a deterministic routing have been used to access a remote memory. In this work, we propose adaptive routing in NUMA interconnect to exploit path diversity in NUMA systems. In particular, we propose a software-based adaptive routing where packet routing paths are changed by modifying the routing table dynamically during runtime - thus, reducing interconnect channel contention and improving overall performance. Adaptive routing does not minimize memory contention but since the interconnect channel can be shared by multiple NUMA nodes and creates contention, adaptive routing maximizes the interconnect bandwidth by load-balancing traffic across different interconnect channels. We provide a proof-of-concept implementation of software-based adaptive routing on AMD Hypertransport-based system and demonstrate performance benefits, including how it is orthogonal to thread and memory schedulers and complements existing OS (or NUMA) schedulers.