Real-Time Dense Depth Estimation Using Semantically-Guided LIDAR Data Propagation and Motion Stereo

In this letter, we present a method for estimating a dense depth map from a sparse LIDAR point cloud and an image sequence. Our proposed method relies on a directionally biased propagation of known depth to missing areas based on semantic segmentation. Additionally, we classify different object boundaries as either occluded or connected to limit the extent of the data propagation. At the regions with large missing point cloud data, we depend on estimated depth using motion stereo. We embed our method on a bounded interpolation strategy which also considers pixel distance, depth difference and color gradient. We then perform an optimization step based on tensor-based TGV-L2 denoising. Our results show that directional propagation and semantic boundary classification can improve the accuracy of interpolation along the edges for different types of objects. Moreover, our motion stereo scheme increases the reliability of extrapolated depth at the regions with large missing point cloud data. Finally, we show that our implementation strategy can achieve reliable results in real time.