Bridging Global Context Interactions for High-Fidelity Pluralistic Image Completion

We introduce PICFormer, a novel framework for
P
luralistic
I
mage
C
ompletion using a trans
Former
based architecture, that achieves both high quality and diversity at a much faster inference speed. Our key contribution is to introduce a
code-shared
codebook learning using a restrictive CNN on small and non-overlapping receptive fields (RFs) for the
local
visible token representation. This results in a compact yet expressive discrete representation, facilitating efficient modeling of
global
visible context relations by the transformer. Unlike the prevailing autoregressive approaches, we proposed to sample all tokens simultaneously, leading to more than 100× faster inference speed. To enhance appearance consistency between visible and generated regions, we further propose a novel attention-aware layer (AAL), designed to better exploit distantly related high-frequency features. Through extensive experiments, we demonstrate that the efficiently learns semantically-rich discrete codes, resulting in significantly improved image quality. Moreover, our diverse image completion framework surpasses state-of-the-art methods on multiple image completion datasets. The project page is available at
https://chuanxiaz.com/picformer/
.