SCORPIO: A Parallel I/O library for Exascale Earth System Models

As high-performance computing (HPC) applications move to higher-resolution model grids and increase the frequency of the simulation output, achieving good I/O performance is critical to overall application performance. HPC applications also need to run on several different platforms for different users and simulations. Achieving good portable I/O performance is a challenging problem, especially considering the wide range of evolving file systems and computer architectures. Application I/O libraries such as ADIOS2, HDF5, PnetCDF, and netCDF have tried to solve this issue by providing high-level I/O abstractions and hiding the complexity of achieving portable I/O performance across computer architectures within the libraries. Nevertheless, applications often require further performance tuning, optimizations, and the ability to choose among these libraries and output file formats to achieve good portable I/O performance at scale across multiple computer architectures and file systems. In this paper we describe the design and implementation of the SCORPIO library, which was created to alleviate this problem for Earth system models. We discuss the features of the library that we leverage to achieve good I/O performance for production configurations of the DOE E3SM Earth system model.