Convolutional ISTA Network with Temporal Consistency Constraints for Video Reconstruction from Event Cameras

Event cameras produce streams of events with high temporal resolution which do not suffer from motion blur. Current deep networks achieve high-quality video reconstruction from events, but most of them are large and difficult to interpret. In this work, we present a solution to this problem by systematically designing a deep network based on sparse representation. First, we investigate the relationship between events and intensity images. The reconstruction problem is then modelled as a sparse coding problem, which can be solved by the iterative shrinkage thresholding algorithm (ISTA). Second, we expand this into a convolutional ISTA network (CISTA) using algorithm unfolding. Finally, we introduce recurrent units and temporal similarity constraints to enhance the temporal consistency (TC) reconstruction of long videos. Results show that our CISTA-TC network achieves high-quality reconstruction compared with state-of-the-art methods, whilst leading to low memory consumption.