Bridging the Gap Between Genericity and Programmability of Dynamic Resources in HPC

Dynamic Resource Management (DRM) allows dynamically changing the resources assigned to a job during its execution. This flexibility can improve productivity in completed jobs per unit of time, resource utilization rate, and energy consumption, among other metrics. Despite its benefits, DRM remains complex to implement, which is one of the reasons it has not yet been widely adopted in production HPC systems. To address this challenge, we design, implement, and evaluate a methodology aimed at improving the programmability of DRM for iterative algorithm-based applications, while maintaining a generic and flexible foundation. Specifically, we interface the Dynamic Management of Resources API (DMR-API), an application-level abstraction layer that simplifies the adoption of dynamic resources—particularly in classical iteration-based HPC applications—with the Dynamic Processes with PSets (DPP) approach, which provides a set of generic design principles for dynamic resource management in high-performance parallel programming models. This integration makes DRM more accessible for iteration-based HPC applications through the DMR-API, while preserving the generality and flexibility of DPP at the system software level to support a wider range of HPC application types. Our results show that DRM can be effectively leveraged in HPC environments with minimal coding effort, unlocking the benefits of dynamic resource allocation for job throughput and system utilization.