Global optimization of neural network weights

Training a neural network is a difficult optimization problem because of the nonconvex objective function. Therefore, as an alternative to local search algorithms, many global search algorithms have been used to train neural networks. However, local search algorithms are more efficient with computational resources, and therefore numerous random restarts with a local algorithm may be more effective than a global algorithm at obtaining a low value of the objective function. This study examines, through Monte-Carlo simulations, the relative efficiency of a local search algorithm to 8 stochastic global algorithms: 2 simulated annealing algorithms, 1 simple random stochastic algorithm, 1 genetic algorithm and 4 evolutionary strategy algorithms. The results show that even ignoring the computational requirements of the global algorithms, there is little evidence to support the use of the global algorithms examined for training neural networks.