Although feature-based methods have been successfully developed in the past decades for the registration of optical images, the registration of optical and synthetic aperture radar (SAR) images is still a challenging problem in remote sensing. In this letter, an improved version of the scale-invariant feature transform is first proposed to obtain initial matching features from optical and SAR images. Then, the initial matching features are refined by exploring their spatial relationship. The refined feature matches are finally used for estimating registration parameters. Experimental results have shown the effectiveness of the proposed method.