A New Decision-Making Approach via Monte Carlo Tree Search and A2C

Monte Carlo Tree Search (MCTS) is a state-of-the-art algorithm suitable for decision-making problem in adversarial complex environments. In this paper, aimed at the challenge of search space being too huge, we lead importance sampling technology into MCTS and build a more efficient algorithm ImS-MCTS, which makes an improvement using probability mapping. Besides, we apply ImS-MCTS to the Advantage Actor-Critic (A2C) framework with deep learning to form a new decision-making approach. The critic uses neural network to approximate the state-value function; the actor is built based on ImS-MCTS, each expanded new leaf is evaluated by the output from neural network and the results of several Monte Carlo rollouts. An advantage function is defined to optimize the neural network, and neural network with updated parameters will significantly improve the strength of ImS-MCTS in the next iteration. The proposed approach has a wide generality in adversarial complex environments. We test our approach in the video game Assault from Atari 2600. Results prove that ImS-MCTS has better performance than traditional MCTS; the combination of A2C framework and deep learning can make the decision-making approach even stronger.