AutoGT: Automatic Generation of Game Trees for Algorithm Benchmarking

Game tree search algorithms play a vital role in application of computational and artificial intelligence methods in games and sequential combinatorial optimization problems. This paper addresses the need for robust and comprehensive benchmarking of these algorithms through the procedural generation of benchmarks. We introduce a novel game tree generator AutoGT built upon the idea of top-down propagation of various game parameters. Thanks to its extensive parameterization, AutoGT enables researchers to evaluate their methods on virtually unlimited number of unique benchmarks. To demonstrate the efficacy and usefulness of the generator, we present comparative results of several standard game-playing algorithms: a game theory optimal player (GTO), a player making moves according to a uniform random distribution, and a few variants of Monte Carlo Tree Search (MCTS) based players. Our findings indicate that for a given set of benchmark parameters, the results are repeatable with low deviation. However, when testing across various parameter settings, the performance becomes more diverse, allowing for a broader and more detailed assessment of the algorithms. The paper includes a link to the full code of AutoGT benchmark generation method.