A semi-direct approach to structure from motion

Reconstructing three-dimensional structure and motion is often decomposed into two steps: point feature correspondence and three-dimensional reconstruction. This separation often causes gross errors since correspondence relies on the brightness constancy constraint that is local in space and time. Therefore, we advocate the necessity to integrate visual information not only in time (i.e. across different views), but also in space, by matching regions-rather than points-using explicit photometric deformation models. We present an algorithm that integrates 2D region tracking and 3D motion estimation into a closed loop based on an explicit geometric and photometric model, while detecting and rejecting outlier regions that do not fit the model. Our algorithm is recursive and suitable for real-time implementation. Our experiments show that it far exceeds the accuracy and robustness of point feature-based SFM algorithms.