Online, Model-Free Motion Planning in Dynamic Environments: An Intermittent, Finite Horizon Approach with Continuous-Time Q-Learning

This paper presents an online kinodynamic motion planning scheme for dynamically evolving environments, by employing Q-learning. The methodology addresses the finite horizon continuous-time optimal control problem with completely unknown system dynamics. An actor-critic structure is employed along with a buffer of previous experiences, to approximate the optimal policy and alleviate the learning signal requirements. The methodology is equipped with a terminal state evaluation to achieve fast navigation. The path planning is assigned to the RRT^X. An obstacle augmentation and a local re-planning strategy are responsible for collision-free navigation. Rigorous Lyapunov-based proofs are provided to guarantee closed-loop stability of the equilibrium point. We evaluate the efficacy of the methodology with simulations.