Bidirectional Hybrid LSTM Based Recurrent Neural Network for Multi-View Stereo

Recently, deep learning based multi-view stereo (MVS) networks have demonstrated their excellent performance on various benchmarks. In this paper, we present an effective and efficient recurrent neural network (RNN) for accurate and complete dense point cloud reconstruction. Instead of regularizing the cost volume via conventional 3D CNN or unidirectional RNN like previous attempts, we adopt a bidirectional hybrid Long Short-Term Memory (LSTM) based structure for cost volume regularization. The proposed bidirectional recurrent regularization is able to perceive full-space context information comparable to 3D CNNs while saving runtime memory. For post-processing, we introduce a visibility based approach for depth map refinement to obtain more accurate dense point clouds. Extensive experiments on DTU, Tanks and Temples and ETH3D datasets demonstrate that our method outperforms previous state-of-the-art MVS methods and exhibits high memory efficiency at runtime.

1 Introduction

Multi-view stereo (MVS), which aims to reconstruct a dense representation for an observed 3D scene from more than two calibrated images, is one of the fundamental issues of computer vision and photogrammetry. Developing general and efficient MVS algorithms has received a wide concern for decades in various applications, such as cultural heritage, urban planning, remote sensing, and autonomous driving.