Learning to Drive in the Open Racing Car Simulator Using Online Neuroevolution

In this paper, we applied online neuroevolution to evolve nonplayer characters for The Open Racing Car Simulator (TORCS). While previous approaches allowed online learning with performance improvements during each generation, our approach enables a finer grained online learning with performance improvements within each lap. We tested our approach on three tracks using two methods of online neuroevolution (NEAT and rtNEAT) combined with four evaluation strategies ( -greedy, -greedy-improved, softmax, and interval-based) taken from the literature. We compared the eight resulting configurations on several driving tasks involving the learning of a driving behavior for a specific track, its adaptation to a new track, and the generalization capability to unknown tracks. The results we present show that, notwithstanding the several challenges that online learning poses, our approach 1) can successfully evolve drivers from scratch, 2) can also be used to transfer evolved knowledge to other tracks, and 3) can generalize effectively producing controllers that can drive on difficult unseen tracks. Our results also suggest that the approach performs better when coupled with online NEAT and also indicate that -greedy-improved and softmax are generally better than the other evaluation strategies. A comparison with typical offline neuroevolution suggests that online neuroevolution can be competitive and even outperform traditional offline approaches on more difficult tracks while providing all the interesting features of online learning. Overall, we believe that this study may represent an initial step toward the application of online neuroevolution in games.