Location Instruction-Based Motion Generation for Sequential Robotic Manipulation

Deep learning-based visuomotor control for manipulation are studied recently, which uses a neural network to learn the mapping from images to robotic actions directly. Most previous methods assume there is only one target object in the image. When multiple target objects are present, they are normally regarded as multi-task problems. One-hot vectors are often used to denote different tasks in the visuomotor framework. Nonetheless, these one-hot vector-based methods are easily disturbed and non-extendable. In this paper, a location instruction is used to guide the robot to generate different trajectories in the situation with multiple targets. The proposed framework mainly consists of three modules: the AutoEncoder (AE) network, the motion generation network and the location detection network. AE tries to process the perception information into small-scale feature maps. The location detection network gives the pixel coordinates of the center of the target object in the image. The motion generation network combines the location instruction and the preprocessed perception information and generates an entire motion trajectory to finish the specified manipulation task. For the object stacking tasks with distractors, the proposed method could obtain a success rate of 98%, while the one-hot vector-based method only has a success rate of 56%.