Development of a 3D Semantic Segmentation Camera based on Mask Regional Convolutional Neural Network

3D semantic segmentation is important for the performance improvement of a robot vision task. Detection and identification of a large variety of objects in a scene are challenging for autonomous robotic manipulation. With the semantic segmented depth information, a robotic arm can obtain the 3D regions of target objects in a complicated scene. In this paper, we present the development of a 3D semantic segmentation camera based on the Mask Regional Convolutional Neural Network (Mask R-CNN) for the understanding of 3D images at the color level. Each object class is assigned by color for representation. It efficiently separates different objects in an image while concurrently produces a high-quality segmentation for each object. This camera is developed with a structured light technique to obtain 3D information with high accuracy and density. The experimental results demonstrate that the 3D information and semantic segmentation can be combined to achieve reliable scene perception.