Automatically Prepare Training Data for YOLO Using Robotic In-Hand Observation and Synthesis

Deep learning methods have recently exhibited impressive performance in object detection. However, such methods needed much training data to achieve high recognition accuracy, which was time-consuming and required considerable manual work like labeling images. In this paper, we automatically prepare training data using robots. Considering the low efficiency and high energy consumption in robot motion, we proposed combining robotic in-hand observation and data synthesis to enlarge the limited data set collected by the robot. We first used a robot with a depth sensor to collect images of objects held in the robot’s hands and segment the object pictures. Then, we used a copy-paste method to synthesize the segmented objects with rack backgrounds. The collected and synthetic images are combined to train a deep detection neural network. We conducted experiments to compare YOLOv5x detectors trained with images collected using the proposed method and several other methods. The results showed that combined observation and synthetic images led to comparable performance to manual data preparation. They provided a good guide on optimizing data configurations and parameter settings for training detectors. The proposed method required only a single process and was a low-cost way to produce the combined data. Interested readers may find the data sets and trained models from the following GitHub repository: github.com/wrslab/tubedet
Note to Practitioners
—The background of this study is a requirement in lab automation – Using robots to arrange randomly placed tubes automatically. Before sending test tubes to an examination machine for gradient tests, humans need to categorize and organize the tubes into specific patterns to fit the machine’s internal design. Employing humans is difficult as the tube arrangement requirements are time-varying. A preferred solution is using robots to replace humans. The robots should have a vision system to detect the tubes and a manipulation system to perform physical arranging actions. They will be used in busy seasons while deployed for other tasks in leisure time. Deep neural networks like YOLO are effective for the tube detection task. However, preparing the training data is challenging and unsuitable for lab end users. Pre-trained neural networks are options but have limited tube detection ability and cannot deal with newly included tube types. The method developed in this work helps solve the training data preparation problem. With its support, the robot can automatically prepare training data that has comparable quality to manually labeled ones in a single-process and low-cost way.