Learning Parallax for Stereo Event-Based Motion Deblurring

Due to the extremely low latency, events have recently been utilized to complement lost information in motion deblurring. Existing approaches largely rely on the perfect pixel-wise alignment between intensity images and events, which usually conflicts with the real world. To tackle this problem, we propose a novel coarse-to-fine framework, named network of event-based motion deblurring with stereo event and intensity cameras (St-EDNet), to recover high-quality images directly from the misaligned inputs that contain both blurry images and the concurrent event stream. Specifically, the coarse spatial alignment of the blurry image and the event stream is first implemented with a cross-modal stereo-matching module without the need for ground-truth depths. Then, a dual-feature embedding architecture is proposed to gradually build the fine bidirectional association of the coarsely aligned data and reconstruct the sequence of the latent sharp images. Furthermore, we build a new dataset with stereo event and intensity cameras (StEIC), containing real-world events, intensity images, and dense disparity maps. Experiments on real-world datasets demonstrate the superiority of the proposed network over state-of-the-art methods. The code and dataset are available at https://mingyuan-lin.github.io/St-ED_web/