Avoiding Pitfalls in Networked Key-Value Store for Tiered Memory

This paper describes the performance pitfalls when using tiered memory for a networked key-value store and our approach to avoiding them. We observe that when receiving data over the network, writing data to tiered memory results in multiple data stagings and repetitive user-kernel crossings. We also observe sudden bursts of I/O operations when allocating memory if the slower memory tier is backed by a DAX file system. We address these challenges through (1) PPF (packet peek and forward) that peeks at the packets in the kernel layer with eBPF and streamlines data placement decisions on tiered memory, and (2) OMA (opportune memory allocator) that moves zeroing off the critical path. Performance evaluation with the prototype shows that the adoption of our design improves IOPS by up to 128%.