A Novel Multi-objective Neural Architecture Search Algorithm via Gaussian Progress Sampling

Multi-objective neural architecture search (NAS) algorithms aim to automatically search the neural architecture suitable for different computing power platforms by using multi-objective optimization methods. The LEMONADE algorithm, which is a representative algorithm of multi-objective NAS algorithms, maintains a population of networks on an approximation of the Pareto front of the multiple objectives, such as predictive performance, number of parameters or FLOPs. To address the irrationality and repeatability of only sampling based on cheap objectives in LEMONADE, we propose a novel multi-objective neural architecture search algorithm via Gaussian Process sampling, dubbed GP-LEMONADE. Meanwhile, to make the sampling process more efficient, we design the online predictor based on Gaussian Process to predict expensive objectives, and sample candidate networks by combining cheap objectives and expensive objectives, so as to ensure the rationality and efficiency of sampling. Experiments show that the GP-LEMONADE algorithm evolves 100 generations and obtains the SOTA model with 3.98% test error. This process only takes 7.38 GPU days, which is 26.75 GPU days shorter than that of LEMONADE. Our methods have improved the performance of the LEMONADE algorithm and ensured the rationality and efficiency of sampling during the evolution, which effectively improving the search efficiency of multi-objective NAS algorithms.