A Multi-Fidelity Genetic Algorithm for Hyperparameter Optimization of Deep Neural Networks

Hyperparameter optimization on Machine Learning models is crucial for their correct refinement. For complex big models (such as Deep Learning models), in which a single training model is supposed to have a very high computational cost, this optimization sometimes becomes unfeasible. Multi-fidelity optimization algorithms are a solution to alleviate this computational cost of optimizing hyperparameters of Deep Learning models. In this scope, we propose GAMF2O, a new multi-fidelity algorithm that relies on a genetic algorithm. This paper clearly defines how to adapt the evolutionary scenario to follow the multi-fidelity approach, and we propose a new scheme to evaluate each individual based on the use of two objectives: the result of the low-fidelity evaluation and the learning capacity, with the use of the latter being novel during the evaluation process. Our experimental section allows us to show how our proposal improves the state-of-the-art in different classification and regression problems.