Saliency weight and local quadrant constraint for visual search

In recent years, the demand for directly visual searching based on images and videos becomes stronger and stronger. Most large-scale image retrieval systems are based on the Bag of Words (BoW) or its variant. For current visual search algorithms which are under the structure of BoW, there are two critical issues should be solved: the quantified visual words may reduce the discriminative power of the local features and the neglect of spatial relationship among local features. To address the problems, we propose a novel method based on saliency and local quadrant constraint. First, we introduce the saliency weights into the quantization stage of BoW. We utilize the image saliency to do inverse document (IDF) weighting. And while generating the histogram expression of image, we count the saliency of features instead of simply counting the number of features. Second, the saliency and the similarity characteristic of deformations in local areas are introduced into our model in the post-processing step to satisfy the constraint in spatial relationship among local features. The operation evaluates the weights of matching features by the saliency of query and candidate images, finds near neighbors of the matching features by a threshold and then estimates whether all the matching features in the local regions follow consistent geometric transformation or not. We evaluate the geometric transformation by the relative quadrant of the matching neighbors to center feature (Local Quadrant Constraint, LQC). Experiments show that the proposed method achieves promising improvement while comparing to other visual search methods. Our methods are well complementary to current visual search methods and give a new idea in modeling the spatial information of image.