Improving model reference control performance using model-free VRFT and Q-learning

This paper proposes the combination of two model-free controller tuning techniques, namely linear Virtual Reference Feedback Tuning (VRFT) and nonlinear state-feedback Q-learning, referred to as a new mixed VRFT-Q learning approach. VRFT is first applied to find the stabilizing feedback controller using only input-output (IO) experimental data from the process in a model reference tracking setting. Reinforcement Q-learning is next applied in the same setting using input-state experimental data collected with a perturbed stabilizing VRFT feedback controller in closed-loop, to ensure good exploration of the state-action space and to avoid data collection under non-stabilizing control. The Q-learning controller is then learned from the input-state data using a batch neural fitted framework. The mixed VRFT-Q learning approach is validated on a case study that deals with the position control of a two-degrees-of-motion open-loop stable Multi Input-Multi Output aerodynamic system. Experimental results show that the Q-learning controllers lead to improved control performance over the initial VRFT controllers.