Model-Based Reinforcement Learning Inspired by Augmented PD for Robotic Control

Reinforcement learning (RL) algorithms have emerged as potent tools for solving sequential decision-making problems. These model-free methods have garnered significant achievements. However, their high sample complexity poses a substantial obstacle, particularly in real-world robotic applications. Consequently, model-based approaches are often preferred to mitigate this complexity, despite their limitations due to inherent model biases in asymptotic performance. Traditional dynamic control methodologies enable the design of precise, compliant, and energy-efficient robotic control. Their primary limitation lies in the requirement for accurate analytical dynamic models. In this research, we address this gap by employing an ensemble of probabilistic neural networks (EPNNs) to learn dynamic models of robots directly from sensor data, integrating system uncertainty considerations. Drawing inspiration from the traditional augmented PD (APD) control method, we propose a novel model-based reinforcement learning (MBRL) algorithm, which employs residual learning for both offline optimization and online planning of the policy. Trajectory tracking experiments are conducted using a Baxter robot in both Pybullet simulation and real-world environments, which demonstrate significant improvements in tracking accuracy with our MBRL method. Furthermore, the robustness of this approach is also validated through various load experiments.