A Job-Aware Decision Method for Hybrid HPC Cluster Scenarios

As an emerging science and technology industry, the development direction and strategic height of high-performance computing has become one of the important symbols of scientific and technological progress nowadays, and also a centralized embodiment of a country's comprehensive strength in science and technology. To cope with the development demand of high performance computing, meeting the resource utilization conditions under multi-dimensional constraints and forming reasonable scheduling are always the most urgent issues. Based on the current development status of hybrid HPC clusters and the problem of resource wastage due to unreasonable scheduling allocation, a job task-aware migration policy method applicable to hybrid HPC cluster environment is proposed in this paper: ECP - Perception-aware decision method, ECP - Perception-aware decision method can target HPC job tasks submitted by users The ECP - Perception-aware decision method can make reasonable load prediction for HPC job tasks submitted by users, and migrate the job tasks to a cluster environment with relatively lower resource consumption or monetary cost based on the prediction result and the perception policy. After comparing several different job task migration models in simulated production environments, the results show that the ECP - Perception-aware decision approach enables HPC job tasks to obtain lower resource consumption or lower monetary cost in a hybrid HPC cluster environment than when running in a single physical HPC cluster or Kubernetes cluster, and also enables the hybrid HPC cluster to resources as well as users' demands are better balanced before, which better solves the multi-objective optimization problem of HPC job task-aware scheduling.