Speeding Up Affordance Learning for Tool Use, Using Proprioceptive and Kinesthetic Inputs

End-to-end learning in deep reinforcement learning based on raw visual input has shown great promise in various tasks involving sensorimotor control. However, complex tasks such as tool use require recognition of affordance and a series of non-trivial subtasks such as reaching the tool, grasping the tool, and wielding the tool. In such tasks, end-to-end approaches with only the raw input (e.g. pixel-wise images) may fail to learn to perform the task or may take too long to converge. In this paper, inspired by the biological sensorimotor system, we explore the use of proprioceptive/kinesthetic inputs (internal inputs for body position and motion) as well as raw visual inputs (exteroception, external perception) for use in affordance learning for tool use tasks. We set up a reaching task in a simulated physics environment (MuJoCo), where the agent has to pick up a T-shaped tool to reach and drag a target object to a designated region in the environment. We used an Actor-Critic-based reinforcement learning algorithm called ACKTR (Actor-Critic using Kronecker-Factored Trust Region) and trained it using various input conditions to assess the utility of proprioceptive/kinesthetic inputs. Our results show that the inclusion of proprioceptive/kinesthetic inputs (position and velocity of the limb) greatly enhances the performance of the agent: higher success rate, and faster convergence to the solution. The lesson we learned is the important factor of the intertwined relationship of exteroceptive and proprioceptive in sensorimotor learning and that although end-to-end learning based on raw input may be appealing, separating the exteroceptive and proprioceptive/kinesthetic factors in the input to the learner, and providing the necessary internal inputs can lead to faster, more effective learning.