Robotic Peg-in-Hole Insertion with Tight Clearances: A Force-based Deep Q-Learning Approach

The automatic execution of contact-rich assembly tasks such as peg-in-hole insertion still remains a challenge in industrial manufacturing automation. Deep reinforcement learning (RL) enables agents to learn complex robotic skills, but requires extensive data collection and relatively long execution times when trained online on the physical hardware. In this paper, a robotic setup and RL implementation are presented, which can learn how to successfully perform the peg-in-hole insertion task. The state vector for our force-based learning approach only consists of force/torque (F/T) signals from the robot tooltip without using any position information. We introduce a deep Q-learning (DQN) framework adapted to the task at hand and gather a training data set with a total of 984 peg insertion attempts and 7,884 experiences on the physical setup. Based on this, we apply an offline learning process to improve efficiency by training a large number of policies with different parameter configurations in short time. Finally, the best model configurations are deployed and evaluated on the physical setup reaching a 100% success rate for the insertion task with 0.2 mm clearance.