Classification Based Metadata Management for HDFS

The way data storage is viewed has been changing consistently. The current trend in data storage is Hadoop which provides a scalable data storage mechanism for storing extremely large amount of data and to handle data intensive scientific applications. It makes use of the MapReduce framework and stores the data in HDFS(Hadoop Distributed File System). In HDFS architecture metadata is handled by the NameNode. In this paper, we propose a novel and efficient mechanism for managing the metadata dynamically by classifying the metadata based on its Importance Factor(If) which is a measure of the data's criticality, frequency of access and the importance of the client using the data. Metadata management is divided into three different techniques based on the importance. To save the amount of metadata from being a constraint on the main memory of the NameNode, the concept of sequence files is employed. This approach leads to more efficient low latency metadata operations, at the same time reduces the bottleneck of the NameNode main memory.