Event-Based Video Frame Extrapolation with Recurrent Neural Networks

Traditional video frame interpolation (VFI) methods do not work well in scenarios with high-speed motion. Using event data from event sensors can mitigate the problem since it provides auxiliary information at a very high-frame rate. Recently, event based VFI methods, that use a combination of frames and event information, have shown better performance than traditional VFI methods. However, most existing event based VFI methods predict intermediate frames between two consecutive RGB frames. This results in additional latency because of the need to wait for the next RGB frame, thus not fully exploiting the capability of event sensor. We introduce a novel approach to perform video frame extrapolation i.e., predict the RGB frame based on a current RGB frame and the subsequent events. This problem is non-trivial since the event data are asynchronous and sparse in the spatial domain and only encode the in-tensity change as a binary value. We use a recurrent neural network based on a gated recurrent unit (GRU) and propose a new training mechanism that exploits the event data to reconstruct RGB frames extrapolated from an initial RGB frame. We perform experiments on publicly available datasets to show the value of the proposed method.