State Definition in Deep Reinforcement Learning: A Smoothness Index Approach

In the pursuit of developing autonomous agents capable of learning optimal behaviors, reinforcement learning has emerged as a foundational frame-work. This paper introduces a novel approach for state definition in deep reinforcement learning systems, with a focus on solving pixel-based control scenarios. The proposed frame-work harnesses the capabilities of Convolutional Variational Auto-Encoders to extract compact and meaningful latent representations from high-dimensional pixel observations. These representations serve as the foundation for constructing an efficient state space for reinforcement learning agents. To further enhance the state construction process, the concept of the Smoothness Index is introduced. Smoothness index acts as a criterion for feature selection within the latent space, facilitating the identification of the most relevant and informative states. Demonstrating the effectiveness of iterative smoothness index-based forward selection, this process notably compresses the observation space while retaining essential information. This compression enables the creation of more efficient reinforcement learning agents. The procedure involves collecting and preprocessing observation data given by the OpenAI Gym car racing environment, encoding observations using a convolutional variational auto-encoder, and iteratively selecting features based on smoothness index. Subsequently, the Proximal Policy Optimization algorithm is employed as a policy-based controller to evaluate the effectiveness of the proposed frame-work. Results indicate that this approach yields a remarkably compact state space that either maintains or enhances reinforcement learning agent performance. This innovative frame-work paves the way for the development of more efficient and adaptive reinforcement learning agents in real-world image-based applications.