Estimating Ratings of Computer Players by the Evaluation Scores and Principal Variations in Shogi

Rating systems are widely used to represent the players' strengths or skills in playing major games such as chess, go, and shogi. We present an alternative method for estimating a player's strength by using the moves played. While popular rating systems including Elo rating are based only on the results of games, i.e., wins, losses, and draws, there are substantial demands for evaluating the quality of the moves themselves instead of the game results. Several studies have been presented that use the evaluation scores obtained via game tree searches using a reference program for just this purpose. However, they incur some problems in terms of the computational costs of running the reference program and also the possible bias of a single or a small number of specific programs. Several weak estimations are integrated in our approach, where the estimations are made by focusing on a reference player playing against a target player, instead of using a few reliable programs as references. We assume a reference player is an ordinary program that can produce an evaluation score and a principal variation for each move. If a target player plays against various kinds of computer players, we can integrate the estimations made by each program in order to yield a stable estimation of the skill set of the target player. Our experiments demonstrated that the proposed method yields a reasonable estimation that represents the average of five estimations, each of which was computed using a hundred randomly selected game records.