Learning of hierarchical control structures

The use of externally imposed hierarchical structures to reduce the complexity of learning control is common. However it is clear that the learning of the hierarchical structure by the machine itself is an important step towards more general and less bounded learning. Presented in this paper is a nested Q-learning technique that generates a hierarchical control structure as the robot interacts with its world. These emergent structures combined with learned bottom-up reactive reactions result in a flexible hierarchical control system.