Multi-Stage Dynamic Cuckoo Filters

Deduplication is a highly efficient data reduction technique to improve storage efficiency and save costs. However, the deduplication performance is severely affected by the limited main memory capacity and the disk access bottleneck for the increasing chunk index lookup. In this paper, we propose Multi-Stage Dynamic Cuckoo Filters (MDCF) to speed up index lookup in deduplication systems. Firstly, MDCF modifies the deletion algorithm of standard cuckoo filter to solve the potential problem that it could not identify whether a cell is empty or not. Secondly, MDCF adjusts the insertion algorithm and expands capacity for the increasing load at bucket level or cuckoo filter level. What is more, MDCF integrates a BitSet and CountingSet for each bucket, which significantly reduces the memory overhead and improves the search efficiency without affecting the false positive rate. The experimental results show that compared with DCF, the query, insertion and deletion efficiency of MDCF are improved by about 55.5%, 20.0% and 21.4% respectively. While MDCF's false positive rate was 49.4% lower than that of DBF. And the data backup throughput can be increased by about 163.0% with MDCF to accelerate index lookup.