Reinforcement Learning-Based Hierarchical Optimal Control of Nonlinear Systems

This paper studies the Stackelberg-Nash game problem of nonlinear systems with three players, in which the leader at the upper layer prioritizes decisions while two followers at the lower layer make decisions based on the responses of the leader and they also constitute a zero-sum game. Depending on the coupling relationship of the three players, we define a performance function for each layer. According to Bellman's principle of optimality, we build two Hamilton-Jacobi-Bellman(HJB) equations and derive optimal policies for three players which form a Stackelberg-Nash equilibrium. An integral reinforcement learning-based policy iteration (PI) algorithm is proposed for online learning of the solutions to HJB equations, which relies only on partial information of the system. We build a critical neural network to implement the PI algorithm. Finally, the effectiveness of the proposed algorithm has been demonstrated through a simulation example.