iBalancer: Load-Aware in-Server Flow Scheduling for Sub-Millisecond Tail Latency

Achieving microsecond-scale tail latency poses an extreme challenge to the conventional architecture of “NIC-OS-Application” in the face of high concurrent requests. Existing kernel-bypass network systems improve this situation significantly. Still, they cannot achieve load-aware in-server requests distribution, which in turn not only harms resource efficiency but, more importantly, beats the goal of squeezing tail latency. This paper proposes iBalancer, an in-server proactive load balancer for the kernel-bypass system, which aggressively handles NIC-side flow scheduling according to the load of threads on the processor-side. Furthermore, we propose a novel metric, “polling time interval (PTI),” to quantify the load of worker threads, which not only indicates utilization of the core bound to the worker thread but also reflects the differences in the processing time of different flows. By scheduling flows according to the metric PTI, iBalancer tends to average the queueing latencies of different flows, such as Set & Get operations for an in-memory key-value store. In addition, by decoupling flow scheduling from packet steering, iBalancer achieves a tail latency aware flow-to-core binding and preserves hardware-based request distribution among cores. The proposed system is evaluated and compared to mTCP and Shenango using two representative microsecond-scale network applications: Memcached KVS and a real-time deep-learning-based financial fraud identification application. Experimental results show that iBalancer can process up to 4.75$ \times $ and 1.55$ \times \ $ higher load over mTCP and Shenango under 500μs 99^th percentile tail latency limit on Memcached. For the financial fraud identification application, iBalancer is able to process 4.56$ \times $ and 1.16$ \times $ higher load than mTCP and Shenango considering 900μs tail latency.