Multimodal 2D/3D Registration for Open Augmented Reality Applications

This work aims to build a multimodal and open Augmented Reality application to enable overlaying videos or 3D graphics on natural objects in a real-time tracking process. The target object is detected within the sequence of images through its local points of interest and their corresponding descriptors. The similarity between the query and the reference image is computed and the homography relating 2D features is determined to produce the projective transformation used for the video registration mode. To add a 3D model into the scene, we estimate the real camera pose and we resolve the transformations connecting the virtual and the real camera reference frames. The obtained results from experiments are accurate and computationally effective. The developed system detects and tracks in real-time markerless objects, then overlays accurately videos or 3D graphics on targets.