Supervised Learning Based HDFS Replication Management System

The Hadoop Distributed File System (HDFS) of Apache Hadoop provides a highly reliable static replication technique for computation, which makes a number of applications to follow Apache Hadoop. However, due to the fact that the access rate of every file is different, maintaining the same replication scheme for every file results in the deterioration of performance. By considering this drawback, this work proposes an approach of using predictive analysis to replicate the data files dynamically. With the help of HACE theorem and supervised learning technique, the utilization of each data file can be predicted to create a corresponding replication strategy. For the remaining low potential files, an erasure code is applied to maintain the reliability. So, this approach improves availability while maintaining the reliability of the data files. Furthermore, the complexity reduction is applied to enhance the effectiveness of the prediction when dealing with Big Data.