Topology-Aware Node Allocation on Supercomputers with Hierarchical Network

Modern high performance computing (HPC) systems or supercomputers typically deploy the job scheduling and resource management (JSRM) system, and a special-purpose topology design in the JSRM system is necessary to guide the node allocation. As physical network distances across the compute nodes differ, the node distribution of the job affects the communication cost. The node selection overhead of the JSRM system and optimal node allocation cannot be reconciled, especially in supercomputers with the hierarchical network. Practically, selectively hiding some physical network details, a simplified topology design in the JSRM system is wise and popular, even though it inevitably increases the communication overheads. The multi-dimensional fat-tree is a promising hierarchical network and tends be widely adopted in future interconnection design for extreme-scale systems. By collecting and analyzing three-year historical job logs from a real production supercomputer with a two-dimensional fat-tree network, we reveal that although the overwhelming majority of jobs are small-size and inherently can fit into a single leaf switch, a large proportion of them still get allocated with nodes located in multiple switches, bringing higher communication overheads. To tackle the issue, four different topology designs that strike a balance between the scheduling overhead and the communication cost are proposed. Extensive evaluations show that all the proposed designs can save communication overheads by allocating nodes close to each other in the physical network, and one of the methods that renumbers the compute nodes achieves the best node allocation while maintaining the shortest node selection time. Like the multi-dimensional fat-tree, many sophisticated networks are also evolved from the traditional fat-tree, our proposed methods can be extended and generalized to other similar hierarchical topologies.