Synthesis of Interacting Model-Based and Model-Free Controllers for Optimal Control

This paper presents a framework that integrates model-based and model-free optimal control methods for continuous-time linear systems. The control design starts with a controller derived from an available nominal model, and is subsequently augmented with a learned (model-free) controller component. We formulate an optimality-based condition that determines the switching instant from the purely model-based controller to the composite controller, which contains both model-based and model-free components. We do not impose any requirements on the model’s precision; the switching condition accounts for real-time model mismatch, ensuring that a more misaligned model results in a faster switching time. Finally, to obtain the model-free component in the composite controller, we employ off-policy reinforcement learning (RL) where trajectory data is used to learn the optimal control augmentation.