Managing Massive Amounts of Small Files in All-Flash Storage

All-flash array is a popular memory device available for use in modern high-performance storage systems. Compared with other types of devices such as DRAM, NVRAM, and EEPROM, flash array combines the best features: shock resistance, low cost, low power consumption, and fast access. Moreover, the ever-increasing density of flash memory has led to a dramatic increase in the capacity, which allows the storage of large volume of data. However, flash memory is not optimal for managing a large number of small files because: 1. the small and random write operation is inefficient in flash memory; 2. massive metadata information occupies a significant portion of the namespace, which is relatively limited or scarce in big data storage systems. This paper introduces a novel approach, hash partitioning-based file compaction (HFC), to improve the efficiency of storing and accessing small files in all-flash storage systems. HFC consists of a file compaction tool and an access interface. The compaction tool merges a group (usually a directory) of small files into a set of "big files" to reduce the metadata required to be maintained in the on-chip memory. The data locality and tree structure of those small files are preserved. The access interface is designed to provide transparent access to the small files in the HFC big files. Experimental results confirm that the proposed method significantly enhances the efficiency of managing massive amounts of small files in flash memory in terms of namespace usage and access speed.