SCUP-HPC: System for Constructing and Utilizing Provenance on High-Performance Computing Systems

Provenance, which documents how data are produced and processed, plays an important role in improving research reproducibility. In computing systems, provenance typically describes a suite of data relationships between simulation and analysis results, programs that generate results, and the input data fed to the programs. Recent research has introduced methods for automatically collecting such data relationships by embedding plugins in the operating system to trace file access and process creation. However, these methods fail to fully describe data processing from the perspective of research reproducibility, particularly in high-performance computing (HPC) systems. To ensure the reproducibility on an HPC system, the provenance must describe the following relationships: 1) executed programs and the corresponding job definition submitted to a workload manager, 2) the program and its source code, and 3) the program and its input/output files. We propose SCUP-HPC, a system for constructing and utilizing data provenance that targets modern HPC systems to support the reproducibility of research activities. We developed a suite of techniques to capture data from the operating system and the workload manager. In addition, we developed a suite of techniques to transform the captured data into provenance by extracting the relationships among programs, files, jobs, and source code from the user’s perspective. We evaluated SCUP-HPC with well-known HPC applications, assessing both the functionality of the constructed provenance and the performance impacts on the applications. The results demonstrate that SCUP-HPC can provide a provenance that supports reproducibility with less than 3% performance overhead.