Combining local class patterns and discovered semantics for image retrieval

Detecting meaningful visual entities (e.g. faces, sky, foliage, buildings, etc.) based on supervised pattern classifiers has become a trend in content-based image retrieval. However, a drawback of the supervised learning approach is the need for manually labeled regions as training samples. We propose a semi-supervised framework to discover local semantic patterns and generate their samples for training with minimal human intervention. Image classifiers are first trained on local image blocks from a small number of labeled images. Then, local semantic patterns are discovered from clustering the image blocks with high classification output. Training samples are induced from cluster memberships for support vector learning to form local semantic pattern detectors. During retrieval, similarities based on local class pattern indexes and discovered pattern indexes are combined to rank images. Query-by-example experiments on 2400 unconstrained consumer photos with 16 semantic queries show that the combined matching approach outperformed the fusion of color and texture features significantly in average precision by 37%.