Learning to Reconstruct Confocal Microscopy Stacks From Single Light Field Images

We present a novel deep learning approach to reconstruct confocal microscopy stacks from single light field images. To perform the reconstruction, we introduce LFMNet, a fully convolutional neural network architecture inspired by the U-Net design. A property of LFMNet is its ability to reconstruct with near confocal accuracy a $112 \times 112 \times 57.6 \,\mu {\rm m}^3$ volume ($1287\times 1287\times 64$ voxels) in $\text{50}\,ms$ given a single light field image of $1287\times 1287$ pixels, thus dramatically reducing 720-fold the capture time compared to confocal scanning of assays at the same volumetric resolution and 64-fold the required storage. To prove its applicability in life sciences, our approach is evaluated both quantitatively and qualitatively on mouse brain slices with fluorescently labeled blood vessels. Because of the drastic reduction in image acquisition time, our setup and method are suitable for imaging highly dynamic and light-sensitive events, and in closed-loop systems, where latency due to the reconstruction time is crucial. We provide empirical analysis of the optical design, of the network architecture and of our training procedure to reconstruct confocal-like volumes for a given target depth range. To train our network, we built a data set of 362 light field images of the mouse brain sample and the corresponding aligned set of 3D confocal scans, which we use as ground truth. The data set will be made available for research purposes.