Joint Deep-Unfolding Optimization Learning for Depth Map Arbitrary-Scale Super-Resolution

Color-guided Depth map Super-Resolution (DSR) based on Convolutional Neural Networks (CNN) is a crucial technology to remedy the defects of mainstream commercial depth cameras and has made significant progress in recent years. Nevertheless, this technology is inevitably facing some huge challenges. Firstly, existing CNN-based DSR methods are designed as black-box network architectures. Secondly, few approaches study single model to achieve arbitrary-scale DSR. Thirdly, due to structural inconsistency between dual-modality, color-guided DSR methods always face texture-copying issue. To this end, we propose a novel joint DSR and high-low frequency decomposition optimization model and this model is unfolded into Deep Arbitrary-Scale Unfolding Network (DASU-Net). DASUNet can achieve robust continuous representation ability by alternately-iterative updating of high-low frequencies and depth features. More importantly, Arbitrary-scale Up-sampling Fusion (AUF) module is proposed to achieve arbitrary-scale up-sampling and dual-modality feature fusion. Specifically, two essential components make up the cores of AUF module including arbitraryscale up-sampling block as well as Feature Enhancement and Multiple Strategies Fusion (FEMSF) blocks. In FEMSF block, color features are first enhanced to highlight its inherentlycorrelated structure with the guidance of depth features, and then the enhanced features are modulated according to different fusion strategies. Furthermore, a fast version of DASU-Net is proposed to fit real-time scenes, named FDASU-Net, which can diminish the runtime by several times for a depth map with a size of 640 × 480 during inference. A large number of experiments can demonstrate that our DASU-Net and FDASUNet can transcend many state-of-the-art DSR methods in terms of several quantitative and qualitative indexes.