Persistence and Recovery for In-Memory NoSQL Services: A Measurement Study

NoSQL systems are deployed as the core components for delivering big data Web services today. With growing main memory capacity, we witness the growing interest and deployment of in-memory NoSQL services (IM-NoSQL), which are designed to maximize the utilization of DRAM for ultra low latency services. To address the volatility of DRAM for in-memory computing services, persistence and failure recovery are important functionality for IM-NoSQL. In this paper we report an extensive measurement study on the performance of persistence and recovery for IM-NoSQL. We evaluate the performance and effectiveness of several common mechanisms used for persistence and recovery in the presence of server crashes, such as snapshot and logging based approaches. Through this study, we are able to answer some of the most frequently asked questions in provisioning of IM-NoSQL services: (i) Can an IM-NoSQL system work effectively when the available memory is insufficient to load the whole dataset? (ii) What is the overhead of maintaining snapshot compared to logging? (iii) How fast an IM-NoSQL system can recover in the presence of failure? And (iv) how does an IM-NoSQL system respond to the different persistence models? We report our comprehensive measurement results on execution, persistence and recovery performance of Redis, a representative implementation of IM-NoSQL services.