Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution

Light field (LF) photography is an emerging paradigm for capturing more immersive representations of the real world. However, arising from the inherent tradeoff between the angular and spatial dimensions, the spatial resolution of LF images captured by commercial micro-lens-based LF cameras is significantly constrained. In this paper, we propose effective and efficient end-to-end convolutional neural network models for spatially super-resolving LF images. Specifically, the proposed models have an hourglass shape, which allows feature extraction to be performed at the low-resolution level to save both the computational and memory costs. To fully make use of the 4D structure information of LF data in both the spatial and angular domains, we propose to use 4D convolution to characterize the relationship among pixels. Moreover, as an approximation of 4D convolution, we also propose to use spatial-angular separable (SAS) convolutions for more computationally and memory-efficient extraction of spatial-angular joint features. Extensive experimental results on 57 test LF images with various challenging natural scenes show significant advantages from the proposed models over the state-of-the-art methods. That is, an average PSNR gain of more than 3.0 dB and better visual quality are achieved, and our methods preserve the LF structure of the super-resolved LF images better, which is highly desirable for subsequent applications. In addition, the SAS convolution-based model can achieve three times speed up with only negligible reconstruction quality decrease when compared with the 4D convolution-based one. The source code of our method is available online.

I. Introduction

As a promising technology for capturing the real-world in a more immersive manner, light field (LF) imaging [1] not only records the accumulated intensity at each image point (i.e., spatial information), but also separates intensity values for each ray direction (i.e., angular information). Alternatively, the resulting LF image implicitly encodes the three-dimensional (3D) geometry information of the scene, which facilitates a wide range of applications, such as image post-refocus [2], depth inference [3], [4], 3D reconstruction [5], virtual/augmented reality [6], to just name a few. Especially, recent advance in commercial hand-held light field cameras, e.g., Lytro Illum [7] and Raytrix [8], opens up the possibility for convenient acquisition of LF images, making research in LF image processing increasingly popular. We refer the readers to [9] for a comprehensive survey on LF imaging and processing.