Incorporating image segmentations into a visual query language for content-based image retrieval

Image retrieval systems have often used color and texture features successfully, but shape features have not been as successful, largely because of the difficulty of specifying them. Traditional image segmentation algorithms tend in many cases to produce results that users find unsatisfactory as either representations of the image or as components of an interpretation of the image. We have found, however, that it is possible to take advantage of the inaccuracy of segmentation to help achieve generality in a query and thus improve recall. Ultimately, the success of these systems will depend on an effective interaction between the user and the system. In order to support this interaction, we are exploring means by which users can specify multiple segmentations for a query image and then edit and annotate the segmentation in order to more accurately express a specific query.