Let's Roll: A Synthetic and Real Dataset for Pedestrian Detection Across Different Shutter Types

Computer vision (CV) pipelines are typically eval-uated on datasets processed by image signal processing (ISP) pipelines. However, in order to deploy CV pipelines in resource-constrained devices, we need to avoid as many ISP steps as possible. In particular, most CV datasets consist of global shutter (GS) images even though most cameras today use a rolling shutter (RS). This paper studies the impact of different shutter mechanisms on machine learning (ML) object detection models using our novel Let's Roll dataset. This dataset consists of paired GS and RS images that we created using a combination of Unreal Engine 5 (UE5) and real snapshots we captured on the streets of downtown Los Angeles, spanning a variety of environments and lighting conditions. More specifically, we trained and evaluated mainstream object detection models with our paired GS and RS images to ascertain whether there exists a significant difference in detection accuracy between these two shutter modalities, especially when capturing low-speed objects (e.g., pedestrians). In particular, we experimented with both YOLOv8 and DETR models to assess the impact of different shutter effects. Our experiments show that well-trained RS-based models can perform similarly to traditional GS-based pipelines, particularly for coarse-grained object detection. Results also show the value of augmenting small real sub-datasets with synthetic images, yielding models with up to a 13.1% improvement in accuracy.