A Weighted Smooth Q-Learning Algorithm

Q-learning and double Q-learning are well-known sample-based, off-policy reinforcement learning algorithms. However, Q-learning suffers from overestimation bias, while double Q-learning suffers from underestimation bias. To address these issues, this letter proposes a weighted smooth Q-learning (WSQL) algorithm. The proposed algorithm employs a weighted combination of the mellowmax operator and the log-sum-exp operator in place of the maximum operator. Firstly, a new stochastic approximation based result is derived and as a consequence the almost sure convergence of the proposed WSQL is presented. Further, a sufficient condition for the boundedness of WSQL algorithm is obtained. Numerical experiments are conducted on benchmark examples to validate the effectiveness of the proposed weighted smooth Q-learning algorithm.