Hybrid Deduplication System—A Block-Level Similarity-Based Approach

Deploying deduplication for primary storage is a challenging task in view of random access patterns of I/O requests and the requirement of quick response time. Existing deduplication approaches designed for primary workloads are locality based with the assumption that I/O requests follow the locality principle. However, primary workloads in cloud systems need not necessarily follow the locality principle, and hence, the existing methods for deduplication are likely to exhibit poor response time. To provide a solution to this challenging problem, we propose and implement a hybrid deduplication system (HDS), a block-based partial deduplication system with similarity-based indexing. The proposed system applies deduplication in the background to decrease the latency and also aims at decreasing the data fragmentation. It applies similarity-based indexing to reduce high number of metadata lookups arising out of random access patterns of the requests. HDS for primary workloads is simulated in the Linux environment using three different types of FIU traces, and the effectiveness of the system is compared with full deduplication based on the parameters-metadata access overhead, average segment length, and response time. The experimental results show that the system has performed consistently better in reducing the metadata overhead and increasing the average segment length for all three sets of I/O trace data.