Improved Optimal Tracking of Uncertain Nonlinear Discrete-time Systems using Experience Replay

This paper addresses the infinite horizon optimal tracking control problem for partially uncertain control-affine nonlinear discrete-time (DT) systems, where the control input dynamics are known. Multi-layer critic and actor neural networks (MNNs) are utilized for online estimation of the infinite horizon value function and optimal control input. The NN weights are tuned online using a direct temporal difference error (TDE)-driven learning approach, which modifies the singular values of the gradient with respect to the NN weights to accelerate their convergence. The critic NN uses a novel experience replay technique to improve sample efficiency without introducing biased TDEs and guarantee the persistence of excitation (PE) condition. The tracking error and weight estimation errors are shown to be uniformly ultimately bounded (UUB) using Lyapunov analysis. The performance of the optimal tracking control scheme with experience replay is evaluated on a two-link robot manipulator and contrasted with model predictive control scheme with known dynamics.