An Erasure Coded Archival Storage System

There is an ever increasing need of storage capacity for storage of digital archives and historical data-digital preservation, because of regulatory and compliance requirements. There is an increasing interest in disk based archival system. Major technical challenges in creating large disk based storage archive are - providing large capacity at low costs, large read and write throughput, data integrity and sustaining hardware and operating system refresh. In this paper we present the architecture and working principle of an archival storage system that uses an erasure-coded redundancy scheme. We present the design of a Quality of Service (QoS) framework that tries to achieve an optimum balance between file availability, performance and system availability. The design includes a file encoding and placement scheme that allows files to be read from the archive without the need to access any metadata. Finally, we present the results obtained from running an experimental setup on Amazon Web Services.