Sarsa-Augmented Off-Policy Reinforcement Learning

In classic online off-policy reinforcement learning (RL), the action sampled from the target policy is used to calculate the temporal-difference target for updating the Q-function. The discrepancy between the data collection policy and target policy can result in certain extrapolation errors that mislead policy evaluation. In this work, the Q-value derived from the State-action-reward-state-action (Sarsa)-style trajectory data is incorporated as a regularization term, resulting in a Sarsa-augmented on-policy regularized policy evaluation (ORPE) approach. Such an operation introduces historical trajectory information, mitigating extrapolation errors and potentially improving stability. The on-policy regularized actor-critic (OR-AC) algorithm is designed by combining ORPE with the maximum entropy actor-critic architecture. We analyze the convergence, conditional monotonicity, and value function gap of ORPE. Comparative experiments on the Mujoco and OpenAI Gym platforms are conducted. The results indicate that the OR-AC algorithm surpasses other mainstream online RL algorithms.