Valve: Scalable RDMA with Gap-based Cache Control Middleware for Data Center Networks

Remote Direct Memory Access (RDMA) has become a cornerstone technology in Data Center Networks (DCNs). However, DCNs have expanded substantially, leading to severe connection scalability issues for RDMA. The critical reason behind these issues stems from frequent cache misses on RDMA NICs (RNICs) when handling numerous Queue Pair (QP) connections. Cache misses require time-consuming retrievals from the host via PCIe, resulting in a degradation in RDMA performance. Existing software solutions primarily aim to alleviate QP Context (QPC) cache pressure, while hardware solutions incur prohibitive costs. In this paper, we identify Work Queue Element (WQE) cache, rather than QPC cache, as the fundamental bottleneck limiting connection scalability. Hence, we propose a software middleware, Valve, designed to mitigate cache misses through WQE cache control. Valve regulates WQE posting to control WQE cache by monitoring RNIC cache usage and adaptively adjusting the gap of WQE posting. Valve boasts ease of deployment, requiring the addition of approximately 1000 lines of code, and incurs low CPU overhead. Valve maintains peak performance of RNIC throughput regardless of the number of QPs and considerably reduces observable cache misses (such as ICM, MTT, and MPT cache misses) by 2.8× to 3.1× compared to XRC and DCT.