PMNet: A Point-to-Mesh Network for 3-D Semantic Instance Reconstruction

Semantic instance reconstruction attracts increasing attention in several areas such as mobile mapping, scene reconstruction, and robot navigation. Although much progresses have been made in recent years, the reconstruction performance is highly sensitive to occlusions and noises. To address these issues, we incorporate point cloud completion into a novel semantic instance reconstruction network PMNet, which consists of a 3-D object detection module, a point cloud completion module, and a mesh generation module. Based on the candidate instance proposals and their proposal features obtained in the object detection module, a point encoder layer is proposed to learn the local geometric features from the point cloud belonging to the detected instances, and a feature transformation layer is utilized to align the proposal features with the local geometric features. These two types of features are then fused and fed into the point cloud decoder to predict the complete point cloud of each instance. The mesh is finally reconstructed for each instance by the mesh generation module. Quantitative and qualitative experiments conducted on the ScanNetv2 dataset demonstrate that the proposed PMNet achieves the best reconstruction performance on real-world point clouds.