Conceptual feedback for semantic multimedia indexing

In this paper, we consider the problem of automatically detecting a large number of visual concepts in images or video shots. State of the art systems involve feature (descriptor) extraction, classification (supervised learning) and fusion when several descriptors and/or classifiers are used. Though direct multi-label approaches are considered in some works, detection scores are often computed independently for each target concept. We propose here a method that we call “conceptual feedback” for improving the overall detection performance that implicitly takes into account the relations between concepts. The vector of normalized detection scores is added to the pool of available descriptors. It is then processed just as the other descriptors for the normalization, optimization and classification steps. The resulting detection scores are finally fused with the already available detection scores obtained with the original descriptors. The feedback of the global detection scores in the pool of descriptors can be iterated several times. It is also compatible with the use of the temporal context that also improves the overall performance by taking into account the local homogeneity of video contents. The method has been evaluated in the context of the TRECVID 2012 semantic indexing task involving the detection of 346 visual or multimodal concepts. Combined with temporal re-scoring, the proposed method increased the global system performance (MAP) from 0.2613 to 0.3014 (+15.3% of relative improvement) while the temporal re-scoring alone increased it only from 0.2613 to 0.2691 (+3.0%).