Pay Attention to How You Drive: Safe and Adaptive Model-Based Reinforcement Learning for Off-Road Driving

Autonomous off-road driving is challenging as unsafe actions may lead to catastrophic damage. As such, developing controllers in simulation is often desirable. However, robot dynamics in unstructured off-road environments can be highly complex and difficult to simulate accurately. Domain randomization addresses this problem by randomizing simulation dynamics to train policies that are robust towards modeling errors. While these policies are robust across a range of dynamics, they are sub-optimal for any particular system dynamics. We introduce a novel model-based reinforcement learning approach that aims to balance robustness with adaptability. We train a System Identification Transformer (SIT) and an Adaptive Dynamics Model (ADM) under a variety of simulated dynamics. The SIT uses attention mechanisms to distill target system state-transition observations into a context vector, which provides an abstraction for the target dynamics. Conditioned on this, the ADM probabilistically models the system’s dynamics. Online, we use a Risk-Aware Model Predictive Path Integral controller to safely control the robot under its current understanding of dynamics. We demonstrate in simulation and in the real world that this approach enables safer behaviors upon initialization and becomes less conservative (i.e. faster) as its understanding of the target system dynamics improves with more observations. In particular, our approach results in an approximately 41% improvement in lap-time over the non-adaptive baseline while remaining safe across different environments.