Where is that pixel in the oblique-view video?

We investigated the problem of deducing the geographical coordinates of pixels in an oblique view video. Our goal is to register the oblique-view video of an urban scene with its cadastral map. The oblique-view videos were taken from a very low flying camera whereas the cadastral map contained only the top-down outline of buildings without any photographic content and without other objects such as trees, cars, people, or any street feature. Our registration comprises a two-step process that uses structure from motion and a matched-filter class of technique. The structure from motion step takes the video as input and outputs a 3D point cloud of the scene. As this point cloud contains objects that are not represented in the cadastral map, our algorithm was designed to emphasize automatically the scene points that are likely to be from building façade so that effects of mismatch in content between the cadastral map and the oblique video can be minimized. For the registration step, we used a coarse-to-fine iterative implementation of matched filter to get a globally optimum solution, with built in tolerance to some scale and rotation invariance. We had implemented the entire system and had tested with real world data. Good results were obtained.