Cloud Data Storage Optimization by Using Novel De-Duplication Technique

Cloud Computing, an internet-based computing is composed of various applications and hardware deployed to the end-user. Existing system deals with block level de-duplication. But in case of block-level de-duplication maintenance of large number of blocks is highly difficult. It also requires high processing power when compared to other de-duplication techniques. This is the reason why File-level De-duplication comes into picture. File based de-duplication is considered due to faster access and easier storage and retrieval of files. This paper proposes to develop a Dynamic cloud storage system and improving the performance of cloud storage by minimizing the data replications using the De-duplication technology - De-duplication Based Cloud Storage (DBCS). In de-duplication, memory occupied by the replicates of the original file in the main server is reduced without removing them from the subsequent servers. For increasing the cloud storage efficiency, de-duplication and compression techniques are adapted. These techniques are to be implemented in the cloud environment via Amazon Web Services (AWS), which is an online web service provider. To develop a dynamic cloud storage system, the operations proposed are: 1. Inline de-duplication using MD5 (client side) 2. Dynamic data reconfiguration 3. Updating resources across servers. The main advantage of using cloud storage from the customers' point of view is that customers can reduce their expenditure in purchasing and maintaining storage infrastructure while only paying for the amount of storage requested, which can be scaled-up and down upon demand. The experiments are performed on different scenarios like 100% popular files, second combination of 30% unpopular files, 70% popular files and third combination of 70% unpopular files, 30% popular files. In all these cases, the proposed approach performs 95% better than traditional techniques in terms of space requirements.