Multiplayer Cascaded Policy Iteration for Nash Differential Games

In this paper, we introduce a method called Multiplayer Cascaded Policy Iteration (MCPI) for finding Nash equilibrium solutions to non-zero-sum (NZS) differential games. While policy iteration (PI) methods have existed for NZS game problems, they usually return solutions lacking stability guarantees during the iterative process due to using a parallel structure in solution-seeking. To tackle such a stability issue, we now propose MCPI with an essentially different structure in series, which is proven to not only guarantee the stability of the iterative process but also exhibit equivalence to quasi-Newton iteration. Based on MCPI, we further develop a state-feedback algorithm to achieve a data-driven solution for multiplayer NZS games, which does not necessitate knowledge of the system dynamics, relying instead on input and state data to learn Nash equilibrium. Furthermore, by leveraging a homotopic method, we provide a path to address a long-standing challenge in PI-based NZS games with completely unknown dynamics, namely, the model demand for an initial stabilizing control policy, which, however, is required in the existing PI-based NZS works. In addition, we broaden the application of the state-feedback algorithm by introducing an output-feedback MCPI algorithm, which alleviates the necessity for full-dimensional measurement of system states. The output-feedback approach deduces optimal solutions for multiplayer NZS game problems and is characterized by its model-free nature, as it leverages input-output data to operate independently of specific knowledge about system dynamics. Finally, we validate the effectiveness of these two model-free algorithms through case studies involving a numerical example and a multi-actuated mechanical rotating rigid body.