Policy Transfer with Maximum Entropy Deep Reinforcement Learning for Real Cart-Pole Control

Transferring learned controllers from simulation to real-world domains is rarely observed for deep reinforcement learning applications, even physics-based tasks like the inverted pendulum being particularly difficult when transitioning to the physical world. Model-free controllers based on deep reinforcement learning have been a successful choice for many tasks. They are preferred over model-based ones due to their adaptive capability to situations not included in the initial design and for requiring less advanced expertise in control. This paper aims to apply learning-based model-free control to a real cart-pole system using a deep reinforcement learning policy transferred from simulation to a real environment without any fine-tuning involved with the dynamics model. In a simulation, the baseline deep Q-learning, an off-policy model-free algorithm, delivered an inferior performance on the Cart-Pole environment compared to the soft Q-learning, keeping the same hyper-parameters and ensuring reproducible outcomes. In the real experimental setup, besides a PD controller feasible for under-actuated systems, the soft version policy was transferred to accelerate the design process and enhance the control. Additionally, limitations related to the brittle robustness of PD controllers and the absence of accurate models in the learning phase are studied and addressed.