Inpainting of Missing Values in the Kinect Sensor's Depth Maps Based on Background Estimates

Low-cost game controllers such as Microsoft's Kinect sensor provide dense, real-time depth measurements of indoor environments at high framerate. The sensor is based on the principle of active stereo using structured infrared light. For surfaces that distract infrared light, no measurements can be obtained. It is important to replace missing values early on, to avoid a slow subsequent conditional evaluations or the propagation of errors into neighboring regions. To solve this problem we present an inpainting method that adds missing values based on background estimates of the unoccluded scene. It is therefore not necessary to hypothesize missing regions based on similarity to other image regions. The procedure also avoids a blurring between foreground and background. By adapting the method to the specific properties of the Kinect (and comparable) cameras, we were able to keep the complexity of the algorithm low, so high speed can be achieved.