An Unsupervised Stitching Method for Light Field Imaging Sensors Using Spatial-Angular Collaborative Representation

Light field (LF) imaging has the ability to capture the distribution of light rays and is widely used in many fields such as depth inference. However, due to the limitations of the optical and sensor structures of LF cameras, the acquired LF images suffer from a narrow field of view (FOV). To this end, this paper proposes a new unsupervised LF stitching method based on spatial-angular collaborative representation, aiming to integrate a set of original LF images acquired from different perspectives to achieve wide-FOV LF reconstruction. Specifically, a spatial warping module is first designed, which combines convolutional neural networks with self-attention to construct a multi-scale feature pyramid. The extracted deep features are used to predict the global homography to achieve overall structural alignment. Correspondingly, the extracted shallow features are responsible for the local deformation adjustment. Secondly, to preserve angular consistency, an angular warping module with dual scanning directions is constructed, which captures the geometric relationship among different views by constructing a pseudo video stack. Based on the above spatial-angular relationship, a bidirectional long short-term memory module is introduced to achieve the collaborative optimization of spatial-angular information, thereby generating alignment results with high spatial-angular coherence. Finally, an encoder-decoder network is leveraged to fuse the alignment results and eliminate parallax artifacts. To validate the performance of the proposed method, a synthetic dataset and a real dataset are established. Experimental results show that the proposed method generates wide-FOV LF images with high spatial-angular quality and outperform existing representative methods in both quantitative and qualitative evaluations.