Joint Discontinuity-Aware Depth Map Super-Resolution via Dual-Tasks Driven Unfolding Network

Convolutional Neural Network (CNN) provides significant performance improvement for Depth map Super-Resolution (DSR). However, CNN-based DSR methods still face two prominent issues: 1) the overall topology structure of CNN cannot be intuitively understood; 2) the boundary prior of depth maps is always not fully utilized or even ignored, which may make the reconstructed depth maps suffer from boundary blurring problem. In this paper, we adopt dual-tasks alternating optimization approach to improve the depth discontinuity-preserving capability of CNN. Concretely, joint optimization problem of two specific tasks is split into two reliant sub-optimization problems, which correspond to two sub-optimization models: DSR sub-optimization model and edge Super-Resolution (SR) sub-optimization model. To achieve interpretability of the CNN, these two sub-optimization models are unfolded into alternatively-optimized DSR sub-network and edge SR sub-network according to proximal gradient descent algorithm, which jointly form the Edge-Constrained DSR Network (EC-DSRNet). The coarse-to-fine architecture of DSR sub-network is designed to gradually capture structures from both color and edge features. Additionally, the proposed joint optimization model makes the proposed network explicitly solve two tasks in an iterative manner, which facilitates an intuitive understanding of the network. A large number of experimental results have demonstrated that the proposed EC-DSRNet can compete against many state-of-the-art DSR. Our codes are publicly released at https://github.com/mdcnn/EC-DSRNet.