A New Approach Towards the Combined Algorithm Selection and Hyper-parameter Optimization Problem

Machine learning algorithms often have many hyper-parameters that can be tuned to improve empirical performance. However, manually exploring the complex search spaces is tedious and cannot guarantee to find satisfactory outcomes. Recently, the Efficient Global Optimization (EGO) algorithm for solving the hyper-parameter optimization problem have shown substantial improvements. However, these algorithms cannot easily be expanded to include the selection of an algorithm. Automatically determining and optimizing an algorithm is known as the combined algorithm selection and hyper-parameter optimization (CASH) problem. In this paper, a novel mixed integer efficient global optimization algorithm and its variants are proposed to solve the CASH problem efficiently. The proposed algorithm is compared with a wide set of state-of-the-art algorithms in both a Black Box Optimization and a CASH problem setting. For the CASH problem setup, seven machine learning algorithms are optimized for a large set of classification t asks. Results show that the proposed algorithms outperform on the black box optimization task and can also outperform alternative state-of-the-art approaches on the CASH problem for specific instances.