HDFSx: An Enhanced Model to Handle Small Files in Hadoop with a Simulating Toolkit

We live in Big Data era, where all data about our lives is captured, stored, processed and used to change the world around us. This data is generated by different sources such as Web, IoT sensors, application server logs, social media, traffic surveillance, and mobile data. The Hadoop distributed file system (HDFS) is established to handle large data files. Handling large number of small files with HDFS is considered a messy process. Thus, HDFSx is a proposed new distribute filesystem architecture that supports big data in small files format. It will enable Hadoop to store and process large number of small files without blocking Namenode memory or flooding network overheads. In order to prove the effectiveness of HDFSx model and algorithms, its performance needed to be evaluated under real scenarios. Implementing such a model in Hadoop environment is very hard, because of inadequacy of Hadoop core classes and libraries documentation which don't have official Class-diagram or Code-Map. In this paper, we propose an open source simulation toolkit for HDFSx called HDFSxSim. It provides a modeling and simulation tool for the main operations of HDFSx model. In addition to, we present a simulation toolkit for HDFS source that could be used to test and evaluate any future modification or adaptation. The experimental results indicated that HDFSx limits the metadata footprint and reduces the network overflow on Namenode.