Is appearance-based structure from motion viable?

Most structure from motion work assumes that tracking is a separate issue from recovery of structure and motion. This paper argues that in certain cases, it helps to use the original images to refine the tracking in the course of estimating structure and motion. We illustrate this by presenting two algorithms: one for appearance-based structure from complete rotation and another for 3-D face modeling from three images. We come to the following conclusion: use appearance prediction only if the gross shape or type of shape deformation is known. Otherwise, one is very likely better off just using the currently known tracks.