A Delayed Container Organization Approach to Improve Restore Speed for Deduplication Systems

Data deduplication has become necessary to improve the space-efficiency of large-scale distributed storage systems, as the global data have accumulated at an exponential rate and they have significant redundancy. However, the negative impact on restore performance is a main challenge for deduplication systems. One of the key reasons is that when restoring data, the low average useful data ratio (UDR) of containers wastes a considerable part of disk bandwidth to read useless data. This is mainly attributed to the uncontrollable compositions of containers. To solve this problem, we propose a new approach called Delayed Container Organization (DCO) to delay the construction of containers after accumulating some redundant data chunks in fast Non-Volatile Memory (NVM) devices to organize high-UDR containers. For example, data chunks in the intersection of some data segments can be organized together in one container to achieve both high deduplication ratio and high UDRs when restoring these related data segments. DCO is implemented in a prototype deduplication system. The experimental results indicate that compared with Capping, DCO promotes the average UDR of containers by 38.30 percent, improves the restore performance by a factor of 2.2, and achieves better space-efficiency and higher cost performance.