Retrieving images by similarity of visual appearance

A system to retrieve images using a description of visual appearance is presented. A multi-scale invariant vector representation is obtained by first filtering images in the database with Gaussian derivative filters at several scales and then computing low-order differential invariants. The multi-scale representation is indexed for rapid retrieval. Queries are designed by the users from an example image by selecting appropriate regions. The invariant vectors corresponding to these regions are matched with those in the database both in feature space as well as in coordinate space, and a match score is obtained for each image. The results are then displayed to the user sorted by the match score. From experiments conducted with over 1500 images of objects embedded in arbitrary backgrounds, it is shown that images which are similar in appearance and whose viewpoint is within 25 degrees of the query image can be retrieved with an average precision of 57.4%