Visual Reinforcement Learning Using Dynamic State Representation for Continuous Motion Control

Visual reinforcement learning (VRL) realizes policy learning based on video recordings of agent-environment interaction processes. However, due to the diversity and uncertainty of pixel information, VRL suffers from low sample efficiency in control tasks. The key challenge lies in finding effective state representation methods to provide effective data samples for policy learning. In this paper, we propose a method for dynamically modeling the state of VRL used for motion control tasks. We first analyze the insufficiency of traditional visual state reconstruction and VRL methods. Then, we introduce a new method that incorporates the four-dimensional spatiotemporal data encoding by 3D convolutional layers for dynamic state representation. Finally, we demonstrate the effectiveness of our proposed method on four representative continuous motion control tasks. Our method effectively enhances the expression of visual states and improves sample efficiency of VRL for continuous motion control.