Markerless augmented reality with a real-time affine region tracker

We present a system for planar augmented reality based on a new real-time affine region tracker. Instead of tracking fiducial points, we track planar local image patches, and bring these into complete correspondence, so a virtual texture can directly be added to them. Moreover, the local image patches can be extracted in an invariant way, even without any a priori information from previous frames. Hence, it is possible to use them as natural beacons, that can be used to recognize the scene and to identify the individual patches. This results in a powerful system that can work without artificial markers or fiducial points and with a minimal amount of user interference.