Relaxation based multiple frame matching of an urban building with street view images

We propose a multi-frame verification of relaxation based matching of clusters of SIFT keypoints. Mean shift clustering is performed to extract groups of SIFT keypoints which are homogeneous on location and orientation as well as gradient vectors. Relaxation based matching significantly enhances the reliability of correspondence, by preserving the structural consistency over matched clusters. The remnant false matches can be removed by verification by multi-frame verification. The consistency level of correspondence over multiple frames is used to distinguish false positives from true positives. Inter-frame homography is used to measure the deviation of the predicted matching vectors from the measured ones. Experiments on matching with Google street view images in San Francisco show that the proposed method achieves significant improvement when more than four consecutive frames are used for verification.