Joint Flow Estimation from Point Clouds and Event Streams

Understanding scene dynamics relies heavily on optical flow and scene flow. Most existing flow estimation methods use low-rate RGB images and point clouds, and match the frames geometrically. However, this approach faces challenges in real-world scenes with intricate motion, occlusion, and noise. To tackle this problem, we combine point clouds with events, which introduce dynamic inter-frame information. We propose a bi-stream neural network that jointly estimates optical flow and scene flow. The event branch extracts dynamic information and estimates optical flow, while the point branch captures scene structure and estimate scene flow. A Spatio-temporal Fusion Block is introduced to fuse the complementary information from points and events. Additionally, we adopt a result-level fusion strategy for direct refinement between the flow predictions of the two branches. We evaluate our model on the real-world datasets DSEC and MVSEC. The experimental results demonstrate superior performance compared to existing methods.