Hierarchic function approximation in kd-Q-learning

Function approximation is an important issue in reinforcement learning for control problems with continuous state space. A new learning algorithm is presented that approximates the quality function with a hierarchic discretization structure called kd-tree. It learns at the beginning for each experienced state transition simultaneously on several hierarchic levels representing different spatial generalizations. As learning proceeds, state transitions get increasingly refined by a descent in the kd-tree scaling down both the spatial and temporal generalization, the latter being the natural abstraction in action space. By increasing the representational complexity within the agent, we can reduce the learning effort considerably.