Resolution Upscaling of 3D Time-of-Flight Sensor by Fusion with RGB Camera

3D time-of-flight (3D ToF) cameras enable depth perception but typically suffer from low resolution. To increase the resolution of the 3D ToF depth map, a fusion approach with a high-resolution RGB camera featuring a new power efficient edge extrapolation algorithm is proposed, implemented and benchmarked here. Despite the presence of artifacts in the output, the resulting high-resolution depth maps exhibit very clean edges when compared to other state-of-the-art spatial upscaling methods. The new algorithm first interpolates the depth map of the 3D ToF camera and combines it with an RGB image to extract an edge map. The blurred edges of the depth map are then replaced by an extrapolation from neighboring pixels for the final high-resolution depth map. A custom 3D ToF and RGB fusion hardware is used to create a new 3D ToF dataset for evaluating the image quality of the upscaling approach. In addition, the algorithm is benchmarked using the Middlebury 2005 stereo vision dataset. The proposed algorithm typically results in a perceived effective upscaling factor in the order of two or more in both the x and y directions.