Two-Branch Filtering Generative Network Based on Transformer for Image Inpainting

Image inpainting has made remarkable progress through deep learning methods. Nevertheless, owing to the flaws of Convolutional Neural Networks (CNNs), the generated contents still contain noticeable artifacts. Some previous studies have aimed to enhance quality by implementing a transformer-based model, but direct substitution is not desirable for inpainting scenarios. To address this issue, we propose a hybrid image inpainting network named Two-branch Filtering Generative Network (TFGN), which combines the predictive filtering technique in CNNs with the self-attention mechanism in the transformer. We specifically incorporate Dual Kernel Filtering Module (DKFM) into an encoder-decoder network. This module utilizes predictive filtering constructed from convolutions to leverage local interactions, while simultaneously employing a transformer architecture with kernels from the predictive network to capture global correlations. Meanwhile, we explore the supplementary effect of structural information and introduce Structure-Aided Kernel Predictive Network (SKPN). SKPN draws on the completed edge map as structural information to supplement the generation of predicted kernels through Base Edge Encoder (BEE). Extensive experiments demonstrate that our model can achieve promising performance and outperforms state-of-the-art methods on various datasets.