Online Synchronous Iterative Algorithm for Optimal Control of Stochastic Affine Nonlinear Systems

In this paper, a synchronous iterative algorithm based on reinforcement learning is proposed for solving the optimal control problem of stochastic affine nonlinear systems subject to state-dependent noise. The algorithm solves the related Hamilton-Jacobi equations online, which are nonlinear and difficult to solve directly, especially when higher-order nonlinear systems are involved. The algorithm is constructed based on an actor/critic structure, which permits the critic and actor neural networks (NNs) formed by value function approximation to be updated synchronously. Persistence of excitation is applied to stochastic nonlinear systems. The author presents an implementation of the algorithm and proves that the NN weights converge to the ideal values. A simulation example validates the effectiveness of the proposed algorithm.