A local-linear-fitting-based matting approach for accurate depth upsampling

We propose an approach for upsampling depth information for RGB color plus depth (RGB-D) images captured with common acquisition systems, where RGB color information is available at all pixel locations whereas depth information is only available at a subset of the pixels. Depth upsampling is formulated as a minimization of an objective function composed of two additive terms: a data fidelity term that penalizes disagreement with the low-resolution observed data and a regularization term that penalizes weighted depth deviations from a local linear model in spatial coordinates, where the weights are determined to ensure consistency between the RGB color image and the estimated depth image. Analogous to techniques used for optimization formulations of image matting, the upsampled depth image is then obtained by solving a large sparse linear system of equations. Visual evaluation of results obtained with the proposed algorithm demonstrate that the method provides high resolution depth maps that are consistent with the color images. Quantitative comparisons demonstrate that the method offers an improvement in accuracy over current state of the art techniques for depth upsampling.