Converging Coevolutionary Algorithm for Two-Person Zero-Sum Discounted Markov Games With Perfect Information

This note proposes a novel population-based coevolutionary algorithm, called the ldquolocal-equilibrium-based coevolutionary algorithmrdquo (LECA), for solving infinite-horizon-discounted two-person zero-sum Markov games with perfect information. LECA ldquorandomizesrdquo a simplified variant of Raghavan and Syed's algorithm, which adapted Howard's policy improvement algorithm for solving Markov decision processes into a ldquonegotiation processrdquo algorithm between two players via a lexicographical search. LECA runs over relatively small sets of policies such that the negotiation process proceeds with the sets via ldquopolicy switching,rdquo rather than with the entire policy spaces of the players. The use of policy switching eliminates the action spaces manipulation in value iteration and policy iteration-type algorithms and derives a novel concept of ldquolocal equilibrium.rdquo With the concept incorporated into the LECA, a ldquolocalrdquo equilibrium policy pair is identified as an ldquoelitist,rdquo and kept in the next population, directing the LECA toward finding an equilibrium or a near-equilibrium policy pair. The algorithm is especially targeted to problems where the state space is small but the action spaces of the players are extremely large, and converges with probability one to an equilibrium policy pair for a given game.