SmartChunk: A hybrid content based chunking algorithm with hash de-duplication for effective data deduplication in cloud storage system

Big Data involves the constant creation and modifying large volumes of data from users all over the world, at all times. Dealing with a substantial amount of data in a live setting can be quite a demanding endeavor. Utilizing a distributed file system is an effective approach for managing and processing substantial amounts of data in real-time scenarios. A distributed file system is a network of separate computers that, to users, functions as a unified and seamless system. In a Distributed file system, common files can be shared between the nodes. However, there are some drawbacks to consider, such as scalability, replication, availability, and the high cost of purchasing a hardware server. Hadoop Distributed File System was developed to address this problem. The Hadoop distributed file system is designed to operate on a cluster of affordable hardware, such as personal computers and laptops. HDFS offers a highly scalable and fault-tolerant storage solution for Big Data, ensuring cost efficiency. The Hadoop Distributed File System ensures high data reliability through data duplication. However, there is a need for more storage space due to the duplication strategy. Efficient management of HDFS storage space can be achieved through the implementation of de-duplication techniques. The goal of the research is to eradicate duplication of files by implementing a De-duplication strategy. In this research work, a novel content based chunking algorithm called SmartChunk is proposed along with data hashing strategy that generates the hash key based on the content of the chunks which will identify if at all any duplicate hash values exists in the prestored hash indices. This will avoid comparing the entire data to identify whether a data already exists there in the cloud or not. The proposed model performance is compared with the existing methods to prove it significance.