Iterative Q-Learning Design for Zero-Sum Games With Evolving Policies

This article aims to achieve data-based online evolving control for zero-sum games with unknown dynamics. First of all, the value-iteration-based Q-learning framework is established. Relevant properties of the iterative Q-learning framework are analyzed, including the convergence and monotonicity. Then, the stability property is investigated and the online data is employed for off-policy learning. More importantly, two effective algorithms are designed to achieve online evolving control. In one algorithm, the monotonically nondecreasing Q-learning sequence requires the admissible criterion to guarantee the stability with the simple Q-function initialization. In another algorithm, the monotonically nonincreasing Q-function sequence can ensure the stability without the admissible criterion, but it requires an elaborate initial Q-function. In the end, by including two examples of real physical backgrounds, the excellent performance of online evolving control is exhibited with the given algorithms.