Adaptive Critic Control With Knowledge Transfer for Uncertain Nonlinear Dynamical Systems: A Reinforcement Learning Approach

This paper presents an online transfer heuristic dynamic programming (THDP) control approach for a class of nonlinear discrete systems. The proposed approach integrates transfer learning with adaptive critic control. To design a robust optimal control strategy for the nonlinear discrete systems, we utilize sample data collected from a source task to acquire prior knowledge. This prior knowledge is subsequently used to guide the online control process of nonlinear systems of target tasks. To avoid negative transfer effects and conserve computational resources, we introduce a novel attenuation function with a truncation mechanism. Additionally, we develop a disturbance compensation control mechanism to address uncertainties. Furthermore, we demonstrate that the properties of the uncertain nonlinear systems under robust optimal control, as well as the weight error of neural networks, are ultimately uniformly bounded given certain conditions. Finally, two simulations are conducted to verify the performance of the proposed algorithm
Note to Practitioners
—Adaptive dynamic programming (ADP) is one of the main methods to solve the Hamilton-Jacobi-Bellman (HJB) equation. However, when using neural network approximation, it often requires a long time of iteration and a large amount of computational process, wasting a lot of computational resources. For this reason, we propose an ADP control scheme with enhanced detection speed: that is, by learning a class of similar tasks to obtain prior knowledge to assist in the online control of our actual system. At the same time, this paper considers system disturbances, which means that they are more universal and robust. After simulation experiments, it has been proven that this scheme has good performance.