Multi-Task Learning of Depth from Tele and Wide Stereo Image Pairs

In this paper, we introduce the problem of estimating the real world depth of elements in a scene captured by two cameras with different field of views, where the first field of view (FOV) is a Wide FOV obtained by a lens with 1 × the optical zoom, and the second FOV is contained in the first FOV and corresponds to a tele zoom lens with 2 × the optical zoom. Traditional stereo matching techniques can estimate the stereo disparity, and hence the depth, in the overlapping FOV between both cameras only, which corresponds to the Tele FOV. We refer to the problem of estimating the disparity, or inverse depth, for the union of FOVs as ‘Tele-Wide disparity estimation’. We propose different deep learning solutions to establish baseline performances. We trained a single-image inverse-depth estimation (SIDE) network to estimate the inverse depth from the image corresponding to the Wide FOV only. We also trained a stereo image disparity estimation network to estimate the disparity for the overlapping Tele FOV only, and another tele-wide stereo matching network (TW-SMNet) for estimating the disparity for the union Wide FOV. We further propose an end-to-end multi-task tele-wide stereo matching deep neural network (MT-TW-SMNet) which attempts to do stereo matching in the overlapped Tele FOV and SIDE in the union Wide FOV. Experimental results on KITTI and the SceneFlow datasets establish baseline performances for the tele-wide stereo matching and demonstrate that multitask tele-wide stereo matching provides a reasonable solution to the Tele-Wide depth estimation problem.