Real-time Dense 3D Semantic Mapping Using RGB-D Camera

Focusing on the poor real-time performance of 3D semantic mapping using RGB-D cameras in indoor environments, this paper investigates the Real-time Dense 3D Semantic Mapping (RD3DSM) using RGB-D camera. Firstly, a lightweight semantic point cloud acquisition algorithm based on keyframe is designed, which is used to segment scene images and obtain the semantic point cloud real-timely. Secondly, to address the loss of inter-frame tracking in the semantic mapping process, ORB-SLAM3 system is employed as the underlying architecture of the visual 3D semantic mapping system to provide stable camera pose. Thirdly, an information fusion mechanism of the visual 3D semantic mapping system is designed, and the OctoMap 3D scene reconstruction module is adopted to implement dense 3D map representation with semantic information, and then a visual 3D semantic mapping algorithm based on ORB-SLAM3 is proposed. Finally, the designed RD3DSM system is integrated into a robot platform. The tests were conducted on the TUM dataset and natural indoor scenes. The results show the effectiveness and real-time performance of the RD3DSM system.