Skill transfer and training in emergent hierarchical control systems

For robots expected to perform tasks of complexity approaching those performed by humans (or other animals) it is becoming clear that robots must be capable of both learning from instruction as well as discovering and learning autonomously. Presented in this paper are methods through which novice robots can be prepared for their future endeavours, not through explicit specification and memorization, but through shaping methods that do not compromise the robot's autonomous learning capabilities. This shaping provides the robot with initial guidance that it is free to use, neglect or alter as it sees fit. Two methods of shaping are discussed: scaffolding actions and staged learning. In general, the robot must be able to transport what it has previously learned (either by chance or in a regimented training program) and apply it to new (possibly related) situations. This paper builds upon recent work in nested Q-learning that allows for the generation of hierarchical control structures and reactive responses in reinforcement learning domains.