Multi-Frame Rolling Shutter Correction With Diffusion Models

The rolling shutter (RS) effect is commonly encountered when capturing images with CMOS sensors, and the RS effect in CMOS sensor imaging frequently results in distortions that degrade image quality and hinder subsequent processing tasks. Traditional rolling shutter correction (RSC) techniques often struggle with complex motion and occlusions. Although deep learning-based methods have advanced the field, they have yet to fully overcome the challenges posed by occlusions. To bridge this gap, we present DM-RSC, a generative framework that utilizes diffusion models (DMs) for multi-frame RSC. Unlike previous works, DM-RSC begins by simultaneously generating bidirectional flow motions between RS frames and their corresponding global shutter (GS) frames. This process not only facilitates the correction of RS frames but also identifies occluded regions. Subsequently, DM-RSC capitalizes on the multi-frame information within occlusion regions and the generative process of DMs to manage occlusion issues, generating the desired GS image. DM-RSC achieves state-of-the-art performance on the leading benchmarks Carla-RS, Fastec-RS, and BS-RSC, marking a leap forward in the domain of RSC. Our code is available at: https://github.com/lhaippp/DM-RSC.