Fast and Scalable Selective Retransmission for RDMA

RDMA, with its high throughput, ultra-low latency, and low CPU utilization, has been widely used in large-scale data centers. However, due to the limited RDMA RNIC on-chip memory, commodity RDMA usually implements the simple Go-Back-N (GBN) loss recovery mechanism, which consumes less memory but leads to a significant performance drop when encountering loss. Recent works try to improve RDMA performance under packet loss by introducing selective retransmission (SR) to it. Nevertheless, implementing efficient SR in RDMA remains challenging. Specifically, either it consumes too much memory for maintaining SR states which leads to poor connection scalability, or it incurs high CPU consumption and latency for onloading SR processing back to the CPU software. To this end, we propose FaSR, a fast and scalable RDMA selective retransmission. It is fast by processing the SR with 200Gbps+ line-rate fully on the RDMA NIC chip, and is scalable by introducing novel SR state management schemes thus consuming small memory even under high concurrency. Specifically, FaSR adopts a dynamically sharing SR structure among connections to reduce the memory footprint by orders of magnitude when the concurrency is high. Also, utilizing the loss recovery pattern, FaSR devises several techniques thus it can access the sharing structure with line-rate for different connections. We have implemented FaSR in Xilinx FPGA board with ~4000 lines of verilog code. Testbed evaluation demonstrates that FaSR can maintain 92%+ throughput at a packet loss rate of 1% under more than 5K concurrent connections, which is 16% and 12.6x higher compared to the latest RDMA SR solution and commodity RDMA NICs, respectively.