Coupling Storage Systems and Self-Describing Data Formats for Global Metadata Management

Traditional I/O stacks feature a strict separation of layers, which provides portability benefits but makes it impossible for storage systems to understand the structure of data. Coupling storage systems with self-describing data formats can offer benefits by making the storage system responsible for managing file metadata and allowing it to use structural information for selecting appropriate storage technologies.Our proposed storage architecture enables novel data management approaches and has the potential to provide significant performance improvements in the long term. By making use of established self-describing data formats, no modifications are necessary to run existing applications, which helps preserve past investments in software development.Specifically, we have designed and implemented an HDF5 VOL plugin to map file data and metadata to object and key-value stores, respectively. Evaluations show that our coupled storage system offers competitive performance when compared with the native HDF5 data format. In some cases, performance could even be improved by up to a factor of 100.