SMA-MVS: Segmentation-Guided Multi-Scale Anchor Deformation Patch Multi-View Stereo

Multi-view stereo aims to recover the 3D model of a scene from a set of images. However, low-textured areas in the scene have always been a challenge in 3D reconstruction. In this work, we propose a segmentation-guided multi-scale anchor deformation patch multi-view stereo. Specifically, we use the Segment Anything Model to distinguish different instances in the scene, and propose an anchor deformation strategy guided by segmentation to adaptively generate a multi-scale anchor patch, so that the depth can be refined from coarse to fine scales. To propagate a better hypothesis for low-textured areas where photometric consistency is unreliable, we propose a non-local adaptive propagation scheme by using the segmented mask as a propagation domain. In order to reduce the interference of illumination on reconstruction completeness, we propose an outlier depth cost refinement guided by reliable points that improves the performance of depth estimation in unevenly illuminated areas. As a result, our method achieves state-of-the-art performance among traditional methods and exhibits better generalization capabilities on the ETH3D, Tanks and Temples, and DTU datasets.