Particle Swarm Optimization and Random Search for Convolutional Neural Architecture Search

Evolutionary and swarm intelligence-based algorithms are commonly used as the search strategy component in Neural Architecture Search (NAS). However, little work has been done to quantify the performance improvements that these nature-inspired metaheuristics offer compared to simpler baseline methods such as random search. This work evaluates the efficacy of Particle Swarm Optimization (PSO) as a NAS strategy for chain-structured Convolutional Neural Networks (CNNs) by conducting thorough and fair comparisons of a PSO-based algorithm (termed evobpso) to 12 alternative methods from the literature as well as random search. A total of 10 benchmark datasets are used for model evaluation, including eight MNIST variations, MNIST-Fashion, and CIFAR-10. The results of this study suggest firstly that evobpso is a competitive NAS algorithm when compared to the literature methods, producing models with the lowest test error rate in three datasets (10.84% in MNIST-RD+BI, 1.62% in MNIST-RB, 5.44% in MNIST-Fashion). Secondly, a statistical comparison of 30 independent executions of evobpso and random search showed that the differences between the mean error rates of the models produced by the two algorithms were rather limited, ranging between 0.02% and 1.9%, but always in favor of evobpso. Therefore, it is concluded that evobpso is a viable NAS strategy able to find top-performing architectures, while random search also merits significant consideration due to its lower complexity and good average performance.