Continuous-Time Spatiotemporal Calibration of a Rolling Shutter Camera-IMU System

The Rolling Shutter (RS) mechanism is widely used in consumer-grade cameras, which are essential parts in smartphones and autonomous vehicles. RS leads to image distortion when the camera moves relative to the scene while capturing images. This effect needs to be considered in structure-from-motion, and vision-aided odometry, for which recent studies have extended Global Shutter (GS) methods to RS cameras. So far, it has been unclear how the RS affects spatiotemporal calibration of the camera in a multi-sensor system, which is crucial for good performance of aforementioned applications. This work takes the camera-IMU system as an example and looks into how the RS affects its spatiotemporal calibration. We develop a continuous-time calibration method for a RS camera-IMU system based on basis splines by using a calibration target. Considering the RS, every observation of a landmark on the target corresponds to a unique camera pose fitted by B-splines. Tests conducted on simulated data generated from public calibration datasets showed that the RS can noticeably affect accuracy of the camera extrinsic parameters, causing errors about 0.5° in orientation and 2 cm in translation with a setting common to smartphone cameras. With real data collected by two industrial camera-IMU systems, we found that considering the RS effect gives more accurate and consistent spatiotemporal calibration. These tests also revealed the impact of alternative IMU models and of the continuous-time versus discrete-time representation on camera extrinsic calibration, and the degradation of visual inertial odometry by extrinsic calibration errors.