Orthogonal Matching Pursuit for Least Squares Temporal Difference with Gradient Correction

The task of reinforcement learning in large or infinite state spaces with linear value function approximation is considered. Feature selection can help improving function approximation accuracy and reducing overfitting. One approach to the problem of feature selection is greedy selection such as orthogonal matching pursuit TD (OMP-TD). However, OMP-TD would perform slow convergence and occasionally unstable behavior for large discount factors. We study the least squares temporal difference with correction method in orthogonal matching pursuit (OMP-TDC). This paper aims to combine advantages of LS-TDC and OMP to derive an efficient policy evaluation. We compare our method against OMP-TD, LS-TDC and LSTD, across several benchmark domains.