Cyclic Exploration and Exploitation in Surprise Minimizing Reinforcement Learning

This paper addresses the exploration-exploitation dilemma in Surprise Minimizing Reinforcement Learning (SMiRL) by introducing a dynamic scaling mechanism for balancing intrinsic and extrinsic rewards. SMiRL encourages agents to visit predictable states to improve learning stability. However, its pre-specified intrinsic weight can excessively bias the policy toward exploitation, leading to inadequate exploration of potentially valuable but initially unpredictable states. To alleviate this issue, we propose a novel adaptive weighting framework based on a latent amplitude metric that indicates the agent’s behavioral convergence. Our method facilitates exploitation until the agent stably converges to a local optimum. Afterwards, it triggers further exploration to find better optima. This timing is inferred from the smoothness of trajectory updates in a learned latent space. We evaluate the proposed method on MuJoCo tasks, a double inverted pendulum and a quadruped locomotion. The results demonstrate that the proposed method outperforms fixed-weight SMiRL in terms of convergence speed and task-specific performance.