Automatic calibration for mobile cameras by fusing multiple relative and absolute visual cues

This paper presents a method for online tracking of a camera's orientation within a man-made scene. The technique applies to novel mobile applications where live video content from hand-held cameras requires image processing such as temporal stitching, stabilization, augmented reality, or other similar operations. The proposed method fuses relative frame-to-frame measurements from a point feature detector with absolute frame-to-scene measurements extracted from vanishing lines within the background of a man-made scene. To achieve this, we propose the use of a Kalman framework exploiting the complementarity of both visual cues in a robust way. The method assumes minimal pose change between consecutive video frames, and assumes that the scene yields sufficient straight lines in at least one of three orthogonal directions. The key insight is that using point features alone may be insufficient in situations where a foreground object moves by or if there are not enough accurate features to register. Moreover, point features provide only a relative frame-to-frame metric, which results in an accumulated error. On the other hand, using vanishing lines is insufficient as well, because it provides inaccurate information in cases where the camera is oriented along one of the three main directions. The strength and novelty of the method is in fusing both observations to overcome their shortcomings. © 2012 Alcatel-Lucent.