Image Inpainting Network Combining Multi-granularity Discrimination and U-transformer

In order to solve the problems of semantic loss and texture blurring that are commonly found in current image inpainting, and to make the inpainting results more in line with the practical needs, we propose an image inpainting network that combines multi-granularity discrimination and U-transformer. After the model first extracts features using a 7x7 convolutional kernel, the representation of these features is enhanced using ConvMod. Then Sequential Polarized Self Attention is introduced and the channel information of the input features is adaptively organized and weighted before other operations to improve the feature representation of the model. Next, a U-transformer structure is used, in which the encoder processes the image by convolution and Transformers to reduce the resolution and extract features; the decoder recovers the dimensions by up-sampling and fusion, and finally outputs the RGB image by convolution. Finally, we introduce a multi-granularity discriminator to evaluate the consistency of the inpainted image with the original image both globally and locally. The results of end-to-end experiments on CelebA, Places2 and Paris StreetView datasets show that the inpainting results of the model are more consistent with visual semantics than the classical approach, and can be inpainted down to the detail texture as well as local features.