Deep Stereo Fusion: Combining Multiple Disparity Hypotheses with Deep-Learning

Stereo matching is a popular technique to infer depth from two or more images and wealth of methods have been proposed to deal with this problem. Despite these efforts, finding accurate stereo correspondences is still an open problem. The strengths and weaknesses of existing methods are often complementary and in this paper, motivated by recent trends in this field, we exploit this fact by proposing Deep Stereo Fusion, a Convolutional Neural Network capable of combining the output of multiple stereo algorithms in order to obtain more accurate result with respect to each input disparity map. Deep Stereo Fusion process a 3D features vector, encoding both spatial and cross-algorithm information, in order to select the best disparity hypothesis among those proposed by the single stereo matchers. To the best of our knowledge, our proposal is the first i) to leverage on deep learning and ii) able to predict the optimal disparity assignments by taking only as input cue the disparity maps. This second feature makes our method suitable for deployment even when other cues (e.g., confidence) are not available such as when dealing with disparity maps provided by off-the-shelf 3D sensors. We thoroughly evaluate our proposal on the KITTI stereo benchmark with respect state-of-the-art in this field.

1. Introduction

Stereo matching aims at inferring depth by determining corresponding points in images taken by two or more cameras sensing the same scene. To this end several approaches have been proposed and a quite outdated, yet exhaustive, reviewed and evaluation on a small and unrealistic dataset was proposed in [29]. Despite the research efforts in this field, with the introduction of more challenging datasets such as KITTI [6], [7], [24] and Middlebury 2014 [30] it is clear that even the most accurate approaches such as [41] are still far from correctly solving the correspondence problem.