Quanta-Slomo: Single Photon Camera Guided 100x Video Frame Interpolation

Video Frame Interpolation (VFI) enhances video quality by computationally increasing video frame rates. Conventional VFI methods rely on simplified motion assumptions (e.g. linear motion) between adjacent frames to generate intermediate frames, which often causes error in highly dynamic scenarios. The Single Photon Avalanche Diode (SPAD) array is an emerging type of quanta image sensor that can capture videos at very high frame rates but can be heavily corrupted by Poisson noise. We present Quanta-SloMo, a novel VFI method that leverages the high frame rate but noisy video from a SPAD sensor to guide the VFI of high quality, low frame rate video captured by a CMOS sensor. Our framework utilizes residual learning, feature alignment through deformable convolution, multi-frame merging, and handles the noise-to-blur trade-off between SPAD video frames through a virtual exposure stack. We demonstrate that Quanta-SloMo outperforms state-of-the-art VFI methods by significant margins, especially for scenes with large motion.