Snapshot Compressive Imaging Using Domain-Factorized Deep Video Prior

Snapshot compressive imaging (SCI) aims at efficiently capturing high-dimensional data (e.g., multi-spectral images and videos) using a two-dimensional detector, which is a hardware-friendly data acquisition paradigm. However, because of the complex structure of videos (such as the dynamic background and moving foreground), it is challenging to reconstruct a video from the captured measurement. Existing model-based methods for video SCI reconstruction are inadequate to reconstruct the complex structure of videos, and existing supervised deep learning-based methods are with poor adaptability to videos in real scenarios. Inspired by the physically interpreted video decomposition, we suggest an unsupervised video SCI reconstruction method with tailored deep video prior and affine transformation, namely, FactorDVP-T. Our FactorDVP-T infers the parameters of the neural networks and the underlying structure of the original video from the captured measurement using a non-reference loss function in an unsupervised manner. Under FactorDVP-T, a video is first factorized into the moving foreground and static background. The background is further factorized into temporal bases and spatial coefficients, where each factor can be modeled individually using the designated unsupervised networks in FactorDVP-T. Moreover, to tackle the dynamic background in real scenarios, we integrate the affine transformation into FactorDVP-T. Benefiting from the expressive power of unsupervised networks embedded in the physically interpreted video decomposition framework, our methods can reconstruct the videos more effectively and better adapt to various videos in real scenarios, as compared with the model-based methods and supervised deep learning-based methods respectively. Extensive experiments on various videos show that our FactorDVP-T can better adapt to different videos, compared with the state-of-the-art model-based and supervised deep learning-based SCI reconstruction methods.