OV-BIS: Open-Vocabulary Boundary Guide Zero-Shot 3D Instance Segmentation

Open vocabulary 3D instance segmentation aims to align 3D instance segmentation results with natural language text, thereby achieving semantic prediction without relying on predefined class labels for specific scenes, which has been widely used in the field of multimedia. Current open vocabulary 3D instance segmentation methods mainly rely on 2D masks provided by various 2D segmentation foundation models. However, in complex scenes, the calculation of 2D masks often struggles to balance over-segmentation of large objects and under-segmentation of small objects. In this paper, we introduce OV-BIS, a novel zero-shot open vocabulary 3D instance segmentation method that leverages instance boundary information to improve 3D semantic segmentation performance. The key insight of our method is that the edge map as 3D boundary projection is suitable for multi-scale tasks and capable of compensating for the weakness of 2D masks in multi-scale adaptability for complex scenes. Our method aggregates multiview edge maps and 2D masks, iteratively guiding the merging of over-segmented point clouds with regions growing to cluster 3D primitives into distinct 3D instances. By projecting 3D instances onto images and using CLIP to calculate semantic features from multiple perspectives with an outliers filter, 3D semantic instance segmentation has been achieved. Experiments on multiple datasets demonstrate the superiority of our method.