Simultaneously Estimating the Fundamental Matrix and Homographies

The estimation of the fundamental matrix (FM) and/or one or more homographies between two views is of great interest for a number of computer vision and robotics tasks. We consider the joint estimation of the FM and one or more homographies. Given point matches between two views (and assuming rigid geometry of the camera-scene displacement), it is well known that all of the matched points satisfy the epipolar constraint that is usually characterized by the FM. Subsets of these point matches may also obey a constraint characterized by a homography (all matches in the subset coming from three-dimensional (3-D) points lying on a 3-D plane). The estimations of homographies and the FM are well-studied problems, and therefore, the (separate) estimation of the FM, or the homography matrices, can be considered as effectively solved problems with mature algorithms. However, the homographies and FM are not independent of each other: therefore, separate estimation of each is likely to be suboptimal. In this paper, we propose to simultaneously estimate the FM and homographies by employing the compatibility constraint between them. This is done by first concentrating on a set of parameters that (jointly) parameterize the entire set of homographies and FM (simultaneously) and that also implicitly enforce the compatibility between the estimates of each set. We then derive a reduced form with the purpose of improving the speed. We propose a solution method in which the Sampson error for the FM and homographies is minimized by the Levenberg-Marquardt (LM) algorithm. Experiments show that the gains can be compared with separate estimates (the FM and/or the homographies).