Reusing Garbage Data for Efficient Workflow Computation

High-performance computing (HPC) systems, including Clusters, Grids and the most recent Clouds, have emerged as attractive platforms to tackle various applications. One significant type of applications in the HPC systems is workflow computation, which has been applied in various scientific and engineering domains. The workflow computation frequently produces intermediate result files, which become garbage after being used and are usually cleaned up without making any contribution to future computation. In this paper, we argue that such garbage data could be useful in the future computation and should not be immediately cleaned up. This is because workflow computation usually contains multiple instances that may share some common data products produced in the past. This sharing scheme provides opportunities to reuse the historical data to speed-up subsequent computation and simplify re-computation due to faulty or crashed runs. To this end, we propose a novel approach, referred to as garbage data manager (GDM), for the workflow computation in HPC systems. The GDM organizes and manages the garbage data for batch schedulers to enhance the performance of subsequent computation. The essence of the GDM is to record the history of computation by constructing a dataflow graph on per instance (run) basis and set up inheritance relationships between the different instances of the same workflow, called run-tree, to achieve the data reuse. Our simulation results demonstrate that exploiting the garbage data is an effective way of improving the workflow computation.