Methodology for GPU Frequency Switching Latency Measurement

The push towards exascale and post-exascale computing in HPC and AI brings together thousands of CPUs and specialized accelerator hardware, making energy optimization crucial as power costs rival system purchase prices. Energy efficiency techniques based on frequency and voltage scaling have been developed and fine-tuned for CPUs, which led to deep understanding of how the CPU hardware behaves under frequency adjustments. In contrast, accelerators, particularly GPUs, have not yet been studied to the same extent in this context.We introduce a methodology to evaluate the latency coupled with accelerator frequency scaling driven by the control CPU (GPU switching latency). The approach employs a minimal, iterative workload that allows statistically distinguishing runtime differences between frequency pairs. It first measures execution times for each frequency and then determines the latency of switching from an initial to a target frequency by tracking runtime changes and repeating measurements to ensure statistical robustness. Finally, the methodology filters out outliers from external factors like driver management or system interruptions. The methodology is implemented in the tool LATEST with support for CUDA accelerators. Evaluated on three Nvidia GPUs – GH200, A100-SXM4, and RTX Quadro 6000 – the analysis reveals significant differences in the switching latency, evaluates optimal frequency change rates, and identifies frequency pairs to avoid due to high overhead.