A cloud based model for deduplication of large data

These days the amount of data that is generated or stored in any system is so high and the amount of storage required is large. By consuming a large number of storage space, the processing speed and the retrieval speed of the data is also slow. Deduplication is the process of identifying redundant data from a large amount of given data before storing them to any physical storage. This way the storage space consumed can be reduced. The concept of deduplication is used in many areas and it is one of the challenging research topics in recent times. In this paper, a cloud based model is proposed for implementing deduplication of a large amount of data available. The model comprises of both the deduplication of data before uploading to the cloud storage and the reverse deduplication of data when downloading the necessary data. The paper also describes the various issues faced during deduplication and the different data structures and algorithms that can be used for effective deduplication.