MVSR3D: An End-to-End Framework for Semantic 3-D Reconstruction Using Multiview Satellite Imagery

Semantic 3-D reconstruction from multiview images is essential for applications such as 3-D city modeling and robot navigation. However, existing methods treat semantic segmentation (SS) and height estimation (HE) as separate tasks, leading to suboptimal reconstruction results. To bridge this gap, we introduce MVSR3D, the first end-to-end framework for semantic 3-D reconstruction using multiview satellite images. MVSR3D employs a dual-stream architecture, consisting of the segmentation branch (MVSAM) based on segment anything model (SAM) and the HE branch based on multiview stereo (MVS). To enhance multiview feature fusion, we propose the epipolar cross attention (ECA) module in the MVSAM branch, which integrates image embeddings primarily along epipolar line to exploit complementary multiview information. Unlike conventional multitask learning approaches, we design dedicated interaction modules—the SAM feature-guided (SAM-FG) module and the elevation-guided sparse prompts generator (EGSPG)—to facilitate multitask interaction and feature fusion. Extensive evaluations on the DFC19 and SpaceNet4 datasets demonstrate that MVSR3D significantly outperforms the state-of-the-art multiview multitask learning (MV-MTL) method, improving the mIoU3 metric at a 2.5-m threshold by 37.09%–45.11%.