A Brief Comparative Study of Metaheuristic Approaches for Hyperparameter Optimization of Machine Learning Model

Machine learning models have been successfully applied in numerous fields. Training a model is the most important aspect of machine learning for its successful application for the problem. To improve the training of a machine learning model thereby improve the performance, the selection of features and setting optimal parameters is crucial. Mainly two kinds of parameters are required to deal with, namely internal and external parameters. Internal parameters are model parameters and configurable such as weights of neural networks and their estimation can be done using data set. The hyperparameters such as learning rate, size of layers, number of layers, loss function etc, are external parameters and its values cannot be determined using the data set and it is not the part of the model. Its estimation can be done by the domain expert or using some trial-and-error techniques until it achieves some acceptable values. However, these techniques are highly time-consuming and cannot ensure the optimal values for these hyperparameters. In recent years different metaheuristic techniques have been applied to determine the optimal values of hyper parameters for machine learning models. In this paper we have conducted a brief comparative study of a few popular metaheuristic approaches applied for the hyperparameter optimization for various machine learning models. In this paper various evaluation measures have been considered for comparative analysis of metaheuristic approaches for hyperparameter optimization for deep learning model.