Zero-Shot Image Inpainting using Pretrained Latent Diffusion Models

The DDNM (Denoising Diffusion Null-space Model), a zero-shot image restoration method, uses a pre-trained diffusion-based image generation model and can be applied to a variety of image restoration tasks without task-specific training. However, since DDNM uses a diffusion model trained by the ImageNet as its backbone, its restoration capability is strongly constrained by the limited number of classes of training images. On the other hand, latent diffusion models (LDMs), which perform diffusion-based generation in latent space, are trained on larger datasets and are known to be able to generate a wide variety of images. One of the challenges in applying LDM to DDNM is that DDNM requires processing that takes advantage of the fact that degradation operators can be represented by linear operators, while LDM uses a nonlinear encoder as preprocessing. In this paper, we focus on the fact that the spatial features of the original image are preserved even in latent space, and propose a method that enables the restoration of a wider variety of images by focusing on inpainting as the image restoration task. Experimental results show that the proposed method is capable of diverse and accurate inpainting without any task-specific training.