Evaluation of Game Tree Search Methods by Game Records

This paper presents a method of evaluating game tree search methods including standard min-max search with heuristic evaluation functions and Monte Carlo tree search, which recently achieved drastic improvements in the strength of Computer Go programs. The basic idea of this paper is to use an averaged win probability of positions having similar evaluation values. Accuracy measures of evaluation values with respect to win probabilities can be used to assess the performance of game tree search methods. A plot of win probabilities against evaluation values should have consistency and monotonicity if the evaluation values are produced by a good game tree search method. By inspecting whether the plot has the properties for some subset of positions, we can detect specific deficiencies in the game tree search method. We applied our method to Go, Shogi, and Chess, and by comparing the results with empirical understanding of the performance of various game tree search methods and with the results of self-plays, we show that our method is efficient and effective.