3D-FlowNet: Event-based optical flow estimation with 3D representation

Event-based cameras can overpass frame-based cameras’ limitations for important tasks such as high-speed motion detection during self-driving cars’ navigation in low illumination conditions. The event cameras’ high temporal resolution and high dynamic range allow them to work in fast motion and extreme light scenarios. However, conventional computer vision methods, such as Deep Neural Networks, are not well adapted to work with event data as they are asynchronous and discrete. A popular approach among state-of-the-art is accumulating the events into 2D image-like matrix. Such 2D-encoding representation is compatible with the traditional convolution neural network but sacrifices the time resolution. In this paper, we present a 3D encoding representation for the event data that can better preserve the temporal distribution of the events compared to the 2D-encoding representation. We then propose 3D-FlowNet, a novel network architecture that processes the 3D input representation with the 3D convolution layer based on the new representation. The 3D features are then flattened into 2D so they can be processed by the 2D convolution layer and generate the optical flow estimation. A self-supervised training strategy is adopted due to the lack of labeled datasets for the event-based camera. Finally, the proposed network is trained and evaluated with the Multi-Vehicle Stereo Event Camera (MVSEC) dataset. The results show that our 3D-FlowNet outperforms state-of-the-art approaches with less training epoch (30 compared to 100 of Spike-FlowNet). The code is released in https://github.com/adosum/3D-FlowNet.