EVI-GPBO: Estimated Variance Integration-Based Gaussian Process Bayesian Optimization

Gaussian process Bayesian optimization (GPBO) based on the upper confidence bound is a method for finding the approximate maximum solution using observations maximizing $E + \beta V^{1/2}$ consisting of the estimated average E and variance V obtained by Gaussian process regression. The exploration weight $\beta $ is the coefficient for adjusting the exploration-exploitation trade-off, which affects search performance. In previous studies, constant value, logarithmic decay, and probabilistic value as the exploration weight have been proposed, and they exhibit good search performance. For better search performance, we propose a method for setting the exploration weight using estimated variance integration (EVI) as a novel meta-heuristics algorithm. Because the integration of the estimated variance V represents the total amount of uncertainty in an entire search space, using it as the exploration weight, we can expect the search performance to be higher than that of previous methods. In this study, an exact analytical solution for integrating the estimated variance in the Gaussian kernel case was provided to realize EVI-GPBO. Experiments on searching for the global optimum using benchmark functions for evaluating approximate search algorithms were conducted to evaluate EVI-GPBO. The results indicate that EVI-GPBO can find optimal solutions with a high percentage compared to the existing methods. In a more realistic setting, an experiment on hyperparameter tuning for a support vector machine was conducted. The results indicate that EVI-GPBO obtained a higher score than the other existing methods. In summary, we conclude that EVI-GPBO is an effective search algorithm.