Minimax Optimal Q Learning With Nearest Neighbors

Markov decision process (MDP) is an important model of sequential decision making problems. Existing theoretical analysis focus primarily on finite state spaces. For continuous state spaces, a recent interesting work (Shah and Xie, 2018) proposes a nearest neighbor Q learning approach. Under the streaming setting, in shich samples are received in a sequential manner, the sample complexity of this method is $\tilde {O}\left ({{\frac {|\mathcal {A}|}{\epsilon ^{d+3}(1-\gamma)^{d+7}}}}\right)$ for $\epsilon $ -accurate Q function estimation of infinite horizon discounted MDP with discount factor $\gamma $ , in which $|\mathcal {A}|$ is the size of the action space. However, the sample complexity is not optimal, and the method is suitable only for bounded state spaces. In this paper, we propose two new nearest neighbor Q learning methods, one for the offline setting and the other for the streaming setting. We show that the sample complexities of these two methods are $\tilde {O}\left ({{\frac {|\mathcal {A}|}{\epsilon ^{d+2}(1-\gamma)^{d+2}}}}\right)$ and $\tilde {O}\left ({{\frac {|\mathcal {A}|}{\epsilon ^{d+2}(1-\gamma)^{d+3}}}}\right)$ for offline and streaming settings respectively, which significantly improve over existing results and have minimax optimal dependence over $\epsilon $ . We achieve such improvement by utilizing samples more efficiently. In particular, the method by Shah and Xie, 2018, clears up all samples after each iteration, thus these samples are somewhat wasted. On the other hand, our offline method does not remove any samples, and our streaming method only removes samples with time earlier than $\beta t$ at time t, thus our methods significantly reduce the loss of information. Apart from the sample complexity, our methods also have additional advantages of better computational complexity, as well as suitability to unbounded state spaces. Finally, we extend our work to the case where both state and action spaces are continuous.

I. Introduction

In nonparametric statistics, optimal rates have been established for various statistical tasks [2], [3], [4], [5], with most of them focusing on identical and independently distributed (i.i.d) data, while problems with non-i.i.d samples are rarely explored. Among these problems, the Markov decision process (MDP) is an important one, which is a stochastic control process that models various practical sequential decision making problems [6], [7], [8], [9], [10]. In MDPs, at each time step, an agent selects an action from a set and then moves to another state and receives a reward. Compared with nonparametric estimation for i.i.d data [2], [3], [4], [5], [11] and MDPs with finite state spaces [12], [13], [14], [15], the design of learning algorithms for MDPs with continuous state spaces faces the following two challenges. Firstly, states, actions, and rewards are received sequentially. In early steps, estimates of the value function are inevitably inaccurate due to limited information. Since later estimates depend on earlier results, estimation errors in the early stages will have a negative impact on later estimates. A proper handling of early steps is thus crucially needed. Secondly, with a continuous state space, states do not appear repeatedly, thus the value function cannot be updated step-by-step as in the discrete state space. It is therefore necessary to design new update rules to use the information from neighboring states.