Kernelized Q-Learning for Large-Scale, Potentially Continuous, Markov Decision Processes

We introduce a novel means of generalizing experi- ences agent experiences for large-scale Markov decision processes. Our approach is based on a kernel local linear regression function approximation, which we combine with Q-learning. Through this kernelized regression process, value function estimates from visited portions of the state-action space can be generalized to those areas that have not yet been visited in a non-linear, non-parametric fash- ion. This can be done when the state-action space is either discrete or continuous. We assess the performance of our approach on the game Su- per Mario Land 2 for the Nintendo GameBoy system. We show thatbetter performance is obtained with our kernelized Q-learning approach compared to linear function approximators for this com- plicated environment. Better performance is also witnessed with our approach compared to other non-linear approximators.