Control Approach Combining Reinforcement Learning and Model-Based Control

Reinforcement learning (RL) methods enable us to construct a learning controller that acquires an appropriate control law tailored to a controlled object through trial and error (exploration). However, the exploring property is problematic when we apply RL methods to real world industrial problems, since it is difficult to maintain the control performance during learning. On the other hand, model-based control methods enable us to construct an appropriate controller based on known information (a mathematical model) of the controlled object. However, achievable control performance depends on the precision of the model and does not improve once the design procedure is finished. As a countermeasure for these issues, we introduce a notion of control approach "LeFiCo" in which a learning controller based on an RL method and a fixed (non-learning) controller are used simultaneously, and we propose a new method included in LeFiCo using an RL-based learning controller and a linear quadratic regulator in parallel for a class of nonlinear adaptive optimal control problems. We also conduct a theoretical analysis of this proposed method from the viewpoint of an RL problem, and we illustrate the validity and effectiveness of our method through numerical simulations.