Optimal control of unknown nonlinear systems via a multi-model-based multi-step reinforcement learning framework

Optimal control of unknown nonlinear systems is challenging due to the absence of the underlying dynamics model. Reinforcement learning (RL) has become an effective framework for such control by optimizing policies with measurement data of the system. However, RL often suffers from (i) low sample efficiency due to the absence of domain knowledge (e.g., dynamics model) and (ii) low learning efficiency due to slow propagation of the reward signal. Model-based RL methods and multi-step RL methods were proposed to address these two challenges, respectively. It is natural to combine these two methods to take advantage of both benefits. However, this combination requires the learned single one-step dynamics model, commonly used in model-based settings, to perform multi-step prediction in a recursive manner, leading to the error accumulation (also known as compounding error) problem. This work presents a multi-model dynamics learning framework to address the error accumulation challenge for general model-based multi-step RL methods by circumventing the recursive prediction. We also present a specific multi-model-based multi-step RL algorithm and validate it on benchmark nonlinear systems. It is shown that the multi-model framework improves multi-step prediction of dynamics and that the presented multi-model-based multi-step RL mostly outperforms model-free, single-model, and one-step counterparts, respectively. We also discuss the limitations of this work and potential future work.