Accurate and Efficient 3D Panoptic Mapping Using Diverse Information Modalities and Multidimensional Data Association

3D Panoptic perception is essential for the understanding of real-world environment and plays an increasingly important role in the field of robotics. However, most existing methods heavily rely on image panoptic segmentation networks to acquire panoptic information of the environment, which is time-consuming and susceptible to interference. In this paper, we propose a novel and efficient panoptic mapping method based on multi-source information. Specifically, to improve the real-time performance of the system, we first apply lightweight object detection and semantic segmentation to extract 2D semantic and instance information from images. Second, a panoptic inference algorithm is designed that fully utilizes multi-source information, including geometry-based and learning-based information, to simultaneously reason about background and foreground objects in the environment. Finally, we take advantage of the scalability of the framework by introducing a multi-object tracking algorithm into the framework, thus providing the temporal information among consecutive frames to the data association module. Based on two popular datasets, extensive comparison experiments are conducted to illustrate the effectiveness of the proposed method. Experimental results show that compared with state-of-the-art panoptic mapping methods, the proposed method achieves superior performance in accuracy, real-timeness and stability. Furthermore, we also evaluate our method in real-world scenarios and CPU-only device to demonstrate the feasibility of its practical deployment.