Label localization by appearance guided graph inferring

Automatically localizing the image labels to the corresponding regions is a challenging but valuable task, which provides detailed semantic information for better image understanding and image retrieval. In this paper, we propose a novel appearance guided graph inferring (AGI) framework for label localization. The framework iterates with two stages: graph inferring and appearance learning. Given the image set, each image is over-segmented into a bag of small patches. In the first step, we adopt graph propagation based method to infer the patch labels collaboratively on the whole image set. A multi-cue graph is constructed for more consistent spatial layout and image label constraints are imposed in propagation. In the second step, SVM classifiers are trained as appearance models by gradually exploiting the inferring results. And then the patch labels are reevaluated by the learned appearance model and feedback to the first step. The global graph propagation and local appearance model complement each other by iteration. Extensive experiments on three public datasets, MSRC-v1, MSRC-v2 and SAIAPR TC-12, demonstrate the encouraging performance of our method in comparison with other baselines.