Exploring the Tradeoff Between Reliability and Performance in HPC Systems

Evaluating the trade-off space between performance and reliability is important for data center operators as part of their supercomputer procurement, planning and acceptance testing. While some simple systems can be modeled with tractable analytic methods, in order to capture with fidelity the interaction among such factors as individual component reliability, processor speeds, checkpointing behaviors and effects, workload characteristics (capacity versus capability), and scheduling policies, just to name a few, a simulation-based approach is required. This paper extends Batsim, a flexible and modular batch scheduling and cluster simulation framework that is built on top of the grid simulation framework, SimGrid. These extensions add a fault model and simulated job checkpoint-restart capability. The enhancements are detailed and experiments are performed using this new capability on varying workloads, both synthetic and real, to evaluate impacts on prospective HPC systems with varying levels of per-node reliability. A basic analytical model is constructed for this trade-off, and contrasted with the experimental results to illustrate the utility of the simulator. It is shown that the toolkit enables one to see not only how much worse a workload will perform on a prospective less reliable system, but also how much larger that system would have to increase in size to achieve the same makespan. This can be used to evaluate tradeoffs and is vitally useful in system procurement. The value of the simulation environment is further demonstrated with a complex, real-world workload that performs differently than expected analytically.