An Exploration of Global Optimization Strategies for Autotuning OpenMP-based Codes

Automatic parameter tuning of parallel codes is ubiquitous in today's HPC environments where the performance portability of said codes is expected to keep pace with the perpetual release of new hardware. With changes in hardware, it is often the case that finding an optimal configuration of these codes is challenging and only further complicated by the high dimensionality or discontinuous topologies of the tuning spaces. Selecting a proper optimization strategy to automatically search these spaces is paramount to minimizing the energy and time spent on exploring sub-optimal configurations. Unfortunately, it is often the case that these optimizers have hyperparameters of their own, which are sensitive and can greatly affect the outcome of quickly converging to, or even finding an optimal code configuration. Much of the existing autotuning literature tends to use particular optimizers without describing their hyperparameter selection, leaving readers to figure out how to configure their optimizer for the best performance. In this work we compare and contrast the popular global optimization strategy of Bayesian Optimization (BO) to two less popular strategies: Particle Swarm Optimization (PSO), and Covariance Matrix Adaptive Evolution Strategy (CMA-ES). We sweep the hyperparameters of these three optimizers in the context of tuning OpenMP hyperparameters of four classic OpenMP programs: BT, FT, HPCG, and Lulesh. Our study compares the long-term search behavior and average time-to-convergence between these three optimization strategies in tuning OpenMP codes. We contribute a detailed study of these strategies and provide deeper insights as to their sensitivities, noting the conditions where each performs well, and hinting at which optimizers require minimal tuning of their hyperparameters for desirable tuning results.