Online solution to the linear quadratic tracking problem of continuous-time systems using reinforcement learning

In this paper, reinforcement learning (RL) is employed to find a casual solution to the linear quadratic tracker (LQT) for continuous-time systems online in real time. Although several RL techniques are developed in the literature to solve the LQ regulator, to our knowledge, there is no rigorous result for using RL to solve the LQ tracker. This is mainly because of the requirement for computing a feedforward term in the tracker control which must be done in a noncausal manner backwards in time. To deal with this noncausality problem, an augmented system composed of the original system and the command generator dynamics is constructed, and an augmented LQT algebraic Riccati equation is derived for solving the LQT problem. In this formulation, one can apply RL techniques to solve the LQT problem, computing the feedforward term and the feedback term simultaneously online in real time. The convergence of the proposed online algorithms to the optimal control solution is verified. To show the efficiency of the proposed approach, a simulation example is provided.