Automatic Hyper-Parameter Tuning for Gradient Boosting Machine

Machine learning and predictive modeling have become widely used in many fields. Utilizing the algorithm without tuning the hyper-parameter can lead to the model cannot perform to its best capabilities. Gradient Boosting Machine (GBM) is one of the tree-based models in which the performance can differ greatly depending on its setting. Tuning hyper-parameters of the model requires background knowledge of the algorithm. Moreover, both the performance and cost of the tuning process need to be kept in consideration. In this paper, we proposed an approach based on Genetic algorithm (GA) in process of GBM tuning. The GA is often used in the optimization problem because of the ability to handle more complex problems. We have implemented the GA variation called hyper-genetic to tune the hyperparameter of GBE and explored the best setting possible of the genetic parameters. In the end, we have compared our best setting with the results from Grid-search, Bayesian optimization, and random approach over four sets of data. Our proposed algorithm has competitive performance and outperforms the other algorithm in the dataset with a high dimension while it required smaller computation time on the most dataset at the optimum point.