An Improved Gaussian Mixture Model Based Hole-filling Algorithm Exploiting Depth Information

Virtual views generation is of great significance in free viewpoint video (FVV) as it can avoid the need to transmit a large volume of video data. An important issue in generating virtual views is how to fill the holes caused by occlusion. Using the Gaussian mixture model (GMM) to generate the background reference image is a commonly used hole-filling method. However, GMM usually has poor performance for sequences with reciprocal motion. In this paper, we propose an improved GMM-based method. To avoid the foreground pixels misclassified as the background pixels, we use depth information to adjust the learning rate in GMM. Foreground pixel is given a smaller learning rate than the background. Further, a refined foreground depth correlation (FDC) algorithm is proposed, which generates the background frame by tracking the change of the foreground depth in the temporal direction. In contrast to existing algorithms, we use a sliding window to obtain multiple background reference frames. These reference frames are then fused together to generate a more accurate background frame. Finally, we adaptively choose the background pixel from the GMM and FDC for hole filling. The experimental results show that subjective gain can be achieved, and significant objective gain can be observed in reciprocal motion sequences.