Automatic Generation and Annotation of Object Segmentation Datasets Using Robotic Arm

Modern robot applications often rely on visual information to perform manipulation tasks in semi-structured environments. A common requirement for such systems is to detect objects in the scene and predict their pose either for building a map, for navigation, or for manipulation purposes. Predicting segmentation masks for the objects is usually helpful for accurate object localization. In robotic manipulation however, the objects of interest can often have complicated geometries or there can be several objects in the scene simultaneously. This makes the manual annotation of an object segmentation dataset especially challenging. In this paper, an automated annotation procedure for object segmentation datasets is presented. Our method uses an industrial robot arm and a camera mounted to the robot’s end effector to collect a set of images showing a scene from multiple viewpoints. Assuming that the end effector transformation and scene geometry are known with respect to the robot’s base frame, the object masks can be computed to each image using the projection model of the camera. Through a real-world implementation of the method, our study demonstrates the accuracy of our automated annotation including a genetic algorithm-based optimization of the camera intrinsic parameters and the transformation between the robot’s flange and the camera frame. Our results suggest, that the proposed automated dataset annotation can be a good alternative to manual labeling. The presented approach introduces a novel application of the digital twin paradigm extending the concept to the field of machine learning dataset generation.