Linearly Solvable General-Sum Markov Games

Linearly solvable Markov decision processes (LSMDPs) are a special class of Markov decision processes (MDPs) in which the optimal value function under an exponential transformation satisfies a linear equation. This model was previously extended to a class of two-player, zero-sum Markov games in which the game's equilibrium value function can similarly be derived from a linear equation. In this work, a new class of linearly solvable n-player, general-sum Markov games is proposed. We show that games in this class have a unique Nash equilibrium, and the equilibrium value functions can be derived from a single system of linear equations. We demonstrate how to approximate discrete-state, discrete-action Markov games outside this class through an embedding process analogous to the way LSMDPs approximate standard MDPs. An off-policy reinforcement learning algorithm for general-sum Markov games, which we call Nash-Z learning, is presented. We empirically demonstrate that Nash-Z learning finds equilibrium policies with low regret and that it finds these solutions orders of magnitude faster than the classic Nash-Q learning algorithm.