Light-Field Reconstruction and Depth Estimation from Focal Stack Images Using Convolutional Neural Networks

Light-field (LF) reconstruction from focal stack images has diverse applications including face recognition, autonomous driving, and 3D reconstruction in virtual reality. It is a large-scale ill-conditioned inverse problem and typically requires regularized iterative algorithms to solve, which can be slow. This paper proposes a non-iterative LF reconstruction and depth estimation method based on three sequential convolutional neural networks (CNNs). The first CNN estimates an all-in-focus image from focal stack images. The second CNN estimates 4D ray depth from the estimated all-in-focus image via the first CNN, and focal stack images. The third CNN refines a Lambertian LF that is rendered using the all-in-focus image and ray depth estimated by the first and second CNNs, respectively. Numerical experiments show that the proposed CNN-based method achieves significantly more accurate and/or faster LF reconstruction, compared to a state-of-the-art sequential CNN using a single image, conventional model-based image reconstruction from a focal stack, and direct regression CNN from a focal stack.