Investigation of Maximization Bias in Sarsa Variants

The overestimation of action values caused by randomness in rewards can harm the ability to learn and the performance of reinforcement learning agents. This maximization bias has been well established and studied in the off-policy Q-learning algorithm. However, less study has been done for on-policy algorithms such as Sarsa and its variants. We conduct a thorough empirical analysis on Sarsa, Expected Sarsa, and n-step Sarsa. We find that the on-policy Sarsa variants suffer from less maximization bias than off-policy Q-learning in several test environments. We show how the choice of hyper-parameters impacts the severity of the bias. A decaying learning rate schedule results in more maximization bias than a fixed learning rate. Larger learning rates lead to larger overestimation. A larger exploration parameter leads to worse bias in Q-learning but less bias in the on-policy algorithms. We also show that a larger variance in rewards leads to more bias in both Q-Learning and Sarsa., but Sarsa is less affected than Q-learning.