Online actor-critic algorithm to solve the approximate optimal adaptive control of continuous-time system with state delay

This paper presents an online policy algorithm for obtaining an approximate solution for optimal control of nonlinear continuous-time system with the state-delay. That is to say, this algorithm learns the solution of HJB equation online in real time. First, a new quadratic cost function over infinite time rang is defined by considering the state-delay. Then, an online adaptive algorithm is used to implement the establishment of critic and actor neural networks, which involves simultaneous adaptation of these two networks. Moreover, the rules for updating actor and critic networks are derived based on the new value function. The convergence of the optimal controller is proved by Lyapunov technique. Meanwhile the stability of the system is ensured. Finally, the validity of this new algorithm is verified by a simulation example.