High-Quality I/O Bandwidth Prediction with Minimal Data via Transfer Learning Workflow | IEEE Conference Publication | IEEE Xplore

High-Quality I/O Bandwidth Prediction with Minimal Data via Transfer Learning Workflow


Abstract:

Providing a high-quality performance prediction has the potential to enhance various aspects of a cluster, such as devising scheduling and provisioning policies, guiding ...Show More

Abstract:

Providing a high-quality performance prediction has the potential to enhance various aspects of a cluster, such as devising scheduling and provisioning policies, guiding procurement decisions, suggesting candidate applications for tuning, and identifying probable scaling and porting challenges. Creating such a prediction for the I/O metrics is still challenging, however, due to the intricate interplay of multiple cluster components, making this an ideal case for machine learning. Nevertheless, achieving the required accuracy level with machine learning calls for a substantial amount of high-quality data, which is often a difficult challenge for most HPC clusters. In this work we explore the use of transfer learning to predict the applications’ I/O bandwidth based on a public dataset. As a result, our experiment can provide an I/O bandwidth prediction for a different cluster comparable to the current state-of-the-art result while employing 100 times less data than needed to construct the base model. Furthermore, we evaluate potential future improvements of the proposed workflow.
Date of Conference: 13-15 November 2024
Date Added to IEEE Xplore: 27 November 2024
ISBN Information:

ISSN Information:

Conference Location: Hilo, HI, USA

Funding Agency:


References

References is not available for this document.