Evolving the MCTS Upper Confidence Bounds for Trees Using a Semantic-Inspired Evolutionary Algorithm in the Game of Carcassonne

Monte Carlo tree search (MCTS) is a sampling best-first method to search for optimal decisions. One popular selection mechanism that has proved to be reliable in MCTS is based on the Upper Confidence bounds for Trees (UCT). This attempts to balance exploration and exploitation. However, some tuning of the MCTS UCT is necessary for this to work well. In this work, we use Evolutionary Algorithms (EAs) to evolve mathematical expressions with the goal to substitute the UCT formula and use the evolved expressions in MCTS. Specifically, we evolve expressions using our proposed semantic-inspired evolutionary algorithm in MCTS (SIEA-MCTS). This is inspired by semantics in Genetic Programming (GP), where the use of fitness cases is seen as a requirement to be adopted in GP. Fitness cases are normally used to determine the fitness of individuals and can be used to compute the semantic similarity (or dissimilarity) of individuals. However, fitness cases are not available in MCTS. We extend this notion by using multiple reward values from MCTS that allow us to determine both the fitness values of individuals and their semantics. We show how SIEA-MCTS is able to successfully evolve expressions that yield better or competitive results compared to UCT. We compare the performance of the proposed SIEA-MCTS against MCTS algorithms, MCTS rapid action value estimation algorithms, three variants of the *-minimax family of algorithms, a random controller, and two more EA approaches. We consistently show how SIEA-MCTS outperforms most of these intelligent controllers in the game of Carcassonne.