On the Effects of Simulating Human Decisions in Game Analysis

The analysis of games or game content, e.g. in the context of AI-assisted game design and search-based PCG, is often based on playthroughs that are generated by simulating human decisions using AI players. With this paper, we hope to encourage more systematic analyses of uncertainties and errors potentially introduced by simulating human decisions in this context. To this end, we conduct a case study on StarCraft II to demonstrate the potential effects of these uncertainties. We construct two usecases from an analysis of existing approaches in research that employ simulations of human decisions through game AI. We are able to demonstrate large impacts of the simulations and finally discuss how the resulting uncertainties can be controlled in future work.