Copy or Not? Reference-Based Face Image Restoration with Fine Details

Reference-guided face restoration can have better identity preservation than non-reference-based methods. However, existing methods can (a) easily produce artifacts, possibly attributable to inefficient facial priors and (b) do not well preserve finegrained facial details crucial for identity, such as freckles, tattoos, and scars. In this work, we propose solutions for these problems. (1) We incorporate a stronger facial prior, generative facial prior (GFP), for reference-based face image restoration. (2) We identify an ambiguity and point out that traditional loss prevents the network from heavily copying facial features from the reference. To address this, we set a new goal and come up with a new loss to realize the new goal. More specifically, when the ground truth and reference are different (e.g., differences in wrinkles, makeup, facial hair, etc.), which one should the output look like? As a simple example, ground truth does not have a mole while reference has one. Traditional loss chose the ground truth, which seems natural, but then the network also learns to ignore reference's facial features; during testing, the network often hesitates. Our new goal is to copy features from the reference as much as possible while maintaining semantic consistency with the degraded input. We propose to use spatial minimum loss and cycle consistency loss to realize the new goal and make the network copy features without hesitation. Using only a single reference image, our proposed method is able to restore highly degraded images while accurately capturing finegrained facial details. To our knowledge, we are the first face restoration framework that is able to restore faces at this granularity. Code and data are available at https://github.com/RefineFIR/RefineFIR.