Data-Driven Visual Picking Control of a 6-DoF Manipulator Using End-to-End Imitation Learning

This paper presents a novel convolutional neural network (CNN) based high-level control architecture that uses deep learning technique to realize autonomous picking control of a six-degree-of-freedom (6-DoF) manipulator using the visual information only. The proposed manipulator control system uses a stereo camera as a measurement device to capture a stereo image of the scene in front of the robot. Then, the proposed CNN-based picking controller uses the captured stereo image as an input to predict the optimal picking control command of the manipulator directly. In the collection of the training dataset, we controlled the manipulator to pick up the object-of-interest (OOI) manually and recorded the stereo images and the corresponding control commands. In the CNN training phase, the supervised end-to-end learning technique is used to learn the mapping between the stereo image observation and the picking control commands of the 6-DoF manipulator. Experimental results show that the proposed end-to-end visual picking control system achieves an average of 70% and 60% success rate in the random single-object and multi-object picking task, respectively.