Interactive Learning System for 3D Semantic Segmentation with Autonomous Mobile Robots

Service robots operating in unfamiliar environments require capabilities for autonomous object recognition and learning from user interactions. However, present semantic segmentation methods, crucial for such tasks, often demand large datasets and costly annotations to achieve accurate inference. In addition, they cannot handle all possible objects or environmental variations without a large additional number of images and annotations. Therefore, this study introduces a learning system for semantic segmentation that combines 3D semantic mapping with interactions between an autonomous mobile robot and a user. We show that the proposed system can: 1) autonomously construct 3D semantic maps using an autonomous mobile robot, 2) improve the prediction accuracy of models pre-trained by supervised and weakly supervised learning in new environments, even without interaction, and 3) more accurately predict new classes of objects with a small number of additional coarse annotations obtained through interaction. Results obtained from experiments conducted in a real-world setting using models pre-trained on the NYU, VOC, and COCO datasets demonstrated an improvement in semantic segmentation accuracy when using our proposed system.