Online calibration and synchronization of cellphone camera and gyroscope

Gyroscope is playing a key role in helping estimate camera rotation during mobile video capture. The fusion of gyroscope and visual measurements needs the knowledge of camera projection parameters, the gyroscope bias and the relative orientation between gyroscope and camera. Moreover, the timestamps of gyroscope and video frames are usually not well synchronized. In this paper, we propose an online method that estimates all the necessary parameters while capturing videos. Our contributions are (1) simultaneous online camera self-calibration and camera-gyroscope calibration based on an implicit extended Kalman filter, and (2) generalization of coplanarity constraint of camera rotation in a rolling shutter camera model for cellphones. The proposed method is able to accurately estimate the needed parameters online with all kinds of camera motion, and can be embedded in gyro-aided applications such as video stabilization and feature tracking.