Layered Hole Filling Based on Depth-Aware Decomposition and GAN-Enhanced Background Reconstruction for DIBR

The Depth-Image-Based Rendering (DIBR) algorithm is pivotal in the advancement of Virtual Reality (VR) and Augmented Reality (AR) technologies due to its capacity to generate virtual views from arbitrary perspectives. Nonetheless, the generation process is often marred by the occurrence of holes due to sharp depth transitions, significantly degrading the quality of the synthesized view. To mitigate this issue, this study introduces a layered hole-filling method to enhance the quality of virtual views. The effectiveness of our proposed method is ensured through three key techniques: Firstly, a depth-aware decomposition method is employed to precisely segregate foreground objects from the background within a reference view. This is achieved by leveraging both the reference image and its corresponding depth map, facilitating accurate instance-level separation of foreground objects. Secondly, a Generative Adversarial Network (GAN)-enhanced background reconstruction method is proposed to generate hole-free target views devoid of foreground objects. Lastly, the integration of Masked 3D Image Warping (M3DIW) and Layered Mergence (LM) algorithms facilitates filling holes with foreground or background textures in a layered manner. Comprehensive experimental results demonstrate the superiority of our proposed method compared to state-of-the-art methods. Notably, our method demonstrates an improvement of 7.5% in mean average Peak Signal-to-Noise Ratio (PSNR) and 1.8% in mean average Structural Similarity Index Measure (SSIM) compared to existing techniques. Additionally, it impressively lowers mean average Learned Perceptual Image Patch Similarity (LPIPS) by 28.8% and significantly reduces mean average Fréchet Inception Distance (FID) by 28.9% for all sequences tested. These results affirm the effectiveness of our approach in enhancing the quality of virtual view synthesis within DIBR applications. Source code is available at https://github.com/threedteam/dibr.