Efficient Evolution of Variational Autoencoders

Despite the extensive successes of deep learning, designing effective neural networks remains an unsolved challenge in deep learning. The complexity of interactions between all the network hyperparameters such as activation type, layer size, number of layers, and connection patterns is far beyond current understanding. As a result, designing an effective neural network can be as much art as it is science. To that end, neural architecture search aims to automate the task of network design; however, many neural architecture search systems are not feasible for the typical research institution due to the high cost of operation. In this work we propose two simple techniques that have a dramatic impact on search cost, achieving as much as a 900% speed-up in per generation training time, with minimal impact on the efficacy of the search algorithm. Similarly, we achieve a 50% reduction in search time without sacrificing any performance.