Depth Generation Network: Estimating Real World Depth from Stereo and Depth Images

In this work, we propose the Depth Generation Network (DGN) to address the problem of dense depth estimation by exploiting the variational method and the deep-learning technique. In particular, we focus on improving the feasibility of depth estimation under complex scenarios given stereo RGB images, where the stereo pairs and/or depth ground-truth captured by real sensors may be deteriorated; the stereo setting parameters may be unavailable or unreliable, hence hamper efforts to establish the correspondence between image pairs via supervision learning or epipolar geometric cues. Instead of relying on real data, we supervise the training of our model using synthetic depth maps generated by the simulator, which deliver complex scenes and reliable data with ease. Two non-trivial challenges, i.e., (i) attaining reasonable amount yet realistic samples for training, and (ii) developing a model that adapts to both synthetic and real scenes arise, whereas in this work we mainly deal with the later one yet leveraging state-of-the-art Falling Things (FAT) dataset to overcome the first. Experiments on FAT and KITTI datasets demonstrate that our model estimates relative dense depth in fine details, potentially generalizable to real scenes without knowing the stereo geometric and optic settings.

I. Introduction

Perceiving the depth of the scene is crucial for robotic tasks such as manipulation [1], [2], auto-pilot [3], [4], and navigation [5]. Although state-of-the-art sensors including Time-of-Flight (ToF) cameras, structured light cameras or Light Detection And Ranging (LiDAR) can serve as the quick solutions for depth sensing, their outputs are relatively sparse and may contain defects for many reasons, which is a well-known issue in the robotics community [6]. Stereo RGB cameras, on the other hand, provide a good alternative. It is probably safe to say that compared with the counterparts, they can adapt to highly dynamic situations because they project no light but merely receiving them passively, and thus will not fail due to missing requisite feedback.