Exploring Constrained Reinforcement Learning Algorithms for Quadrupedal Locomotion

Shifting from traditional control strategies to Deep Reinforcement Learning (RL) for legged robots poses inherent challenges, especially when addressing real-world physical constraints during training. While high-fidelity simulations provide significant benefits, they often bypass these essential physical limitations. In this paper, we experiment with the Constrained Markov Decision Process (CMDP) framework instead of the conventional unconstrained RL for robotic applications. We evaluated five constrained policy optimization algorithms for quadrupedal locomotion using three different robot models. Our aim is to evaluate their applicability in real-world scenarios. Our robot experiments demonstrate the critical role of incorporating physical constraints, yielding successful sim-to-real transfers, and reducing operational errors on physical systems. The CMDP formulation streamlines the training process by separately handling constraints from rewards. Our findings underscore the potential of constrained RL for the effective development and deployment of learned controllers in robotics.

I. INTRODUCTION

The use of Deep Reinforcement Learning (RL) for robotic control is on the rise, revolutionizing the way control policies are created for legged robots and other complex dynamic systems. Particularly, model-free approaches have gained prominence, replacing traditional optimization-based methods. This paradigm shift can be attributed to the high-capacity neural network models, effective model-free algorithms that can solve complex problems, and efficient tools for data-generation (i.e. simulations). As a result, the synthesis of locomotion policies for legged robots has become more straightforward and accessible, as evidenced by the growing number of RL-based controllers in recent literature.