Access optimization for small files in HDFS: Challenges and Solutions

The Hadoop Distributed File System (HDFS) is an open-source distributed file system optimized for data appending and reading, offering portability, high fault tolerance, and scalability. After more than a decade of development, HDFS has become widely used for big data storage, supporting both structured and unstructured data at a massive scale. It also enables a variety of applications, such as complex query analysis, interactive analytics, detailed billing queries, Key-Value read/write operations, and iterative computations. However, with the rapid growth of internet applications and scientific computing, the number of small files—such as images and text—has surged. Since HDFS was originally designed for large file storage, it encounters issues when handling large volumes of small files, including inefficient metadata structures, master server bottlenecks, and network latency. As a result, efficiently storing and accessing large numbers of small files has become a significant challenge for HDFS. This paper introduces the principles and features of HDFS, and summarizes existing solutions for managing massive small files in HDFS, including Hadoop’s built-in approaches and user-customized solutions. It also reviews related research conducted in recent years. Looking forward, with ongoing technological advancements, indexing, networking, and caching technologies are expected to become key strategies for optimizing HDFS for small file storage.