Fast Generation of High-Fidelity RGB-D Images by Deep Learning With Adaptive Convolution

Using the raw data from consumer-level RGB-D cameras as input, we propose a deep-learning-based approach to efficiently generate RGB-D images with completed information in high resolution. To process the input images in low resolution with missing regions, new operators for adaptive convolution are introduced in our deep-learning network that consists of three cascaded modules—the completion module, the refinement module, and the super-resolution module. The completion module is based on an architecture of encoder–decoder, where the features of input raw RGB-D will be automatically extracted by the encoding layers of a deep neural network. The decoding layers are applied to reconstruct the completed depth map, which is followed by a refinement module to sharpen the boundary of different regions. For the super-resolution module, we generate RGB-D images in high resolution by multiple layers for feature extraction and a layer for upsampling. Benefited from the adaptive convolution operators proposed in this article, our results outperform the existing deep-learning-based approaches for RGB-D image complete and super-resolution. As an end-to-end approach, high-fidelity RGB-D images can be generated efficiently at the rate of 22 frames/s.
Note to Practitioners
—With the development of consumer-level RGB-D cameras, industries have started to employ these low-cost sensors in many robotic and automation applications. However, images generated by consumer-level RGB-D cameras are generally in low resolution. Moreover, the depth images often have incomplete regions when the surface of an object is transparent, highly reflective, or beyond the distance of sensing. With the help of our method, engineers are able to “repair” the images captured by consumer-level RGB-D cameras in high efficiency. As the typical deep-learning networks are employed in this approach, the proposed approach fits well with the GPU-based hardware architecture of deep-learning computation—therefore, it potentially can be integrated into the hardware of RGB-D cameras.