Revisiting Swapping in User-Space With Lightweight Threading

Memory-intensive applications, such as in-memory databases, caching systems, and key-value stores, are increasingly demanding larger main memory to fit their working sets. Conventional swapping can enlarge the memory capacity by paging out inactive pages to backend stores. However, existing swapping solutions suffer several performance and compatibility issues, making them unsuitable for high-concurrency and memory-intensive applications. In this article, we redesign the swapping system and propose Lightswap, a high-performance user-space swapping solution that supports paging with both local SSDs and remote memories. First, to avoid kernel involvement, we propose to leverage the extended Berkeley packet filter (eBPF) for handling page faults (PFs) in user space and further eliminate the heavy I/O stack with the help of user-space I/O drivers. Then, we co-design the PF handling with lightweight thread (LWT) scheduling to improve system throughput and reduce the end-to-end PF latency. Finally, we propose a try-catch framework in Lightswap to deal with swap-in errors which have been exacerbated by the scaling in process technology. We implement Lightswap in our production-level system and evaluate it with various benchmarks. Results show that Lightswap achieves scalable PF notification latency (
$4 \mu \text{s}$
under 128 LWTs), reduces the PF handling latency by 3–5 times, and improves the throughput of memcached by more than 40% compared with the state-of-art swapping systems.