EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis

Single image super-resolution is the task of inferring a high-resolution image from a single low-resolution input. Traditionally, the performance of algorithms for this task is measured using pixel-wise reconstruction measures such as peak signal-to-noise ratio (PSNR) which have been shown to correlate poorly with the human perception of image quality. As a result, algorithms minimizing these metrics tend to produce over-smoothed images that lack highfrequency textures and do not look natural despite yielding high PSNR values.,,We propose a novel application of automated texture synthesis in combination with a perceptual loss focusing on creating realistic textures rather than optimizing for a pixelaccurate reproduction of ground truth images during training. By using feed-forward fully convolutional neural networks in an adversarial training setting, we achieve a significant boost in image quality at high magnification ratios. Extensive experiments on a number of datasets show the effectiveness of our approach, yielding state-of-the-art results in both quantitative and qualitative benchmarks.

1. Introduction

Enhancing and recovering a high-resolution (HR) image from a low-resolution (LR) counterpart is a theme both of science fiction movies and of the scientific literature. In the latter, it is known as single image super-resolution (SISR), a topic that has enjoyed much attention and progress in recent years. The problem is inherently ill-posed as no unique solution exists: when downsampled, a large number of different HR images can give rise to the same LR image. For high magnification ratios, this one-to-many mapping problem becomes worse, rendering SISR a highly intricate problem. Despite considerable progress in both reconstruction accuracy and speed of SISR, current state-of-the-art methods are still far from image enhancers like the one operated by Harrison Ford alias Rick Deckard in the iconic Blade Runner movie from 1982. A crucial problem is the loss of high-frequency information for large downsampling factors rendering textured regions in super-resolved images blurry, overly smooth, and unnatural in appearance (c.f. Fig. 1, left, the new state of the art by PSNR, ENet-E). Figure 1.

Comparing the new state of the art by PSNR (enet-e) with the sharper, perceptually more plausible result produced by enet-pat at 4x super-resolution on an image from imagenet.