Grave Artifacts in Image Inpainting: Investigating the Causes and Untangling the Factors

Image Inpainting, the task of recovering missing information in an image, is of great practical interest to users who wish to recover damaged photographs or remove unwanted parts of images. Advances in Convolutional Neural Networks and Adversarial Networks have produced large gains in this problem. Nevertheless, Inpainting algorithms may fail to recover a decent image, and even produce jarring artifacts. In this work, we seek to uncover what causes such issues. We investigate the relationship between the artifacts and the type of degradation mask, the size of the mask, and the algorithms used. To that end, we employ rectangular, free-form and segmentation-based degradation masks, of varying sizes. We find that large, continuous regions of missing information are often to blame for the most egregious failures, and not really how complex the shape of the region is. Moreover, we observe that no algorithm is completely immune to these failure modes.