Optimizing the Performance of Consistency-Aware Deduplication Using Persistent Memory

Block-level data deduplication is a widely used technology for saving storage space by filtering the data blocks with the same hash value. However, existing block-level data deduplication approaches either ignore the data consistency of deduplication or suffer severe performance degradation for providing consistency guarantees. In this article, we propose Consistency-Aware Deduplication (CADedup
⁺
) to achieve high-performance block-level data deduplication with data consistency. The main idea of CADedup
⁺
is to achieve an efficient journaling mechanism for deduplication by taking advantage of persistent memory (PM), such as byte-addressability and near-DRAM access latency. To balance the tradeoffs between performance and consistency requirements in data deduplication, we carefully design three modes of journaling mechanism, i.e., writeback mode, ordered mode, and journal mode, for CADedup
⁺
. We properly place the deduplication metadata of CADedup
⁺
onto the DRAM- PM hybrid memory architecture to minimize PM costs according to the features of metadata updates. The deduplication metadata on PM is managed by a set of metadata transactions and updated with the help of the efficient hardware atomic operations provided by CPU. We implement CADedup
⁺
in the generic block layer in Linux kernel 4.9.0. We conduct extensive experiments on Intel Optane PMEM to evaluate CADedup
⁺
with typical benchmarks. Experimental results show that CADedup
⁺
can reduce 63%–70% write volume and 50%–60% I/O latency over Dmdedup, a widely used open-source block-level data deduplication system, while ensuring deduplication consistency.