Intensity Guided Depth Upsampling Using Edge Sparsity and Super-Weighted $L_0$ Gradient Minimization

Although depth cameras acquire depth in dynamic scenes, the captured depth images are often noisy and of low resolution. Depth images have the physical nature of being represented by smooth regions and edges in between them, i.e. depth images have high edge sparsity in the gradient domain. In this paper, we propose intensity guided depth upsampling using edge sparsity and super-weighted L
₀
gradient minimization. First, we get mutual structure between intensity and depth using joint mutual structure filtering. Second, we generate an initial depth image using recursive interpolation. Next, we generate weights for L
₀
gradient minimization based on gradient and entropy of the intensity image, and upsample the depth image using super-weighted L
₀
gradient minimization. In super-weighted L
₀
gradient minimization, we combine two main terms: Hybrid data fidelity and weighted L
₀
gradient regularization. The hybrid data fidelity term combines both zero-order and first-order data differences to suppress staircase artifacts, while the weighted L
₀
gradient regularization term preserves depth structures and removes noise. Finally, we further refine the depth image using adaptive fast weighted median filtering. Experiments on Middlebury and realworld scene datasets verify that the proposed method produces edge preserving depth upsampling results and outperforms state-of-the-arts in terms of both visual quality and quantitative measurements.