An asymmetric real-time dense visual localisation and mapping system

This paper describes a dense tracking system (both monocular and multi-camera) which each perform in real-time (45Hz). The proposed approach combines a prior dense photometric model with online visual odometry which enables handling dynamic changes in the scene. In particular it will be shown how the technique takes into account large illumination variations and subsequently improves direct tracking techniques which are highly prone to illumination change. This is achieved by exploiting the relative advantages of both model-based and visual odometry techniques for tracking. In the case of direct model-based tracking, photometric models are usually acquired under significantly greater lighting differences than those observed by the current camera view, however, model-based approaches avoid drift. Incremental visual odometry, on the other hand, has relatively less lighting variation but integrates drift. To solve this problem a hybrid approach is proposed to simultaneously minimise drift via a 3D model whilst using locally consistent illumination to correct large photometric differences. Direct 6 dof tracking is performed by an accurate method, which directly minimizes dense image measurements iteratively, using non-linear optimisation. A stereo technique for automatically acquiring the 3D photometric model has also been optimised for the purpose of this paper. Real experiments are shown on complex 3D scenes for a hand-held camera undergoing fast 3D movement and various illumination changes including daylight, artificial-lights, significant shadows, non-Lambertian reflections, occlusions and saturations.