A Universal Deduplication Architecture for Secure and Efficient Cloud Storage

Users now produce data at a rate that exceeds their ability to securely store and manage it all, provoking them to entrust their private files to Cloud Storage Providers (CSPs). These companies discreetly inspect users’ files to undertake deduplication, which stores only a single instance of files that are redundant across their user base. By undertaking deduplication in this way, the CSP acquires low-cost storage at the expense of user privacy. This paper proposes universal deduplication, an alternative approach which shifts the advantage of deduplication from the CSP to the users, while ensuring semantic security of the users’ transmitted data. Universal deduplication leverages indications of the trustworthiness of data availability on the Internet, paired with a format to automatically combine client-side deduplication and end-to-end encryption. By referencing data that is publicly available on the Internet, user files can be privately deduplicated without the need to transmit sensitive user data, while simultaneously reducing storage and encryption costs. An architecture for the implementation of universal deduplication is proposed in this paper, along with a preliminary investigation into the feasibility of the proposed concepts.