Connectivity Schemas in NeuroEvolution: What Neural Architectures does GEPNN evolve?

In recent years, there has been a rise in popularity of using evolutionary algorithms (EA's) in conjunction with artificial neural networks (ANN's). This approach is commonly known as NeuroEvolution (NE). NeuroEvolutionary approaches typically optimize just the weights of an ANN or optimize the architecture, learning rates, thresholds, and weights together. Algorithms capable of the latter are known as Topological and Weight Evolving ANN (TWEANN). One such TWEANN is Gene Expression Programming for Neural Networks (GEPNN). This paper presents an empirical investigation of the network topologies that arise when GEPNN is used and whether evolved architectures have any relation to state of the art architectures. Results show that GEPNN naturally discovers powerful structural motifs such as shortcut connections and also creates sparse networks. Both these schemas have been shown to be advantageous in deep learning techniques. As an additional contribution from this work, we provide an open source library for developing GEPNN solutions in Python.