Emergent proximo-distal maturation through adaptive exploration

Life-long robot learning in the high-dimensional real world requires guided and structured exploration mechanisms. In this developmental context, we investigate here the use of the recently proposed PI
_CMAES
²
episodic reinforcement learning algorithm, which is able to learn high-dimensional motor tasks through adaptive control of exploration. By studying PI
_CMAES
²
in a reaching task on a simulated arm, we observe two developmental properties. First, we show how PI
_CMAES
²
autonomously and continuously tunes the global exploration/exploitation tradeoff, allowing it to re-adapt to changing tasks. Second, we show how PI
_CMAES
²
spontaneously self-organizes a maturational structure whilst exploring the degrees-of-freedom (DOFs) of the motor space. In particular, it automatically demonstrates the so-called proximo-distal maturation observed in humans: after first freezing distal DOFs while exploring predominantly the most proximal DOF, it progressively frees exploration in DOFs along the proximo-distal body axis. These emergent properties suggest the use of PI
_CMAES
²
as a general tool for studying reinforcement learning of skills in life-long developmental learning contexts.