Beyond Accuracy: Typicality Ranking for Video Annotation

In this paper, we address the issue of typicality ranking for video annotation and propose to use a novel criterion, Average Typicality Precision (ATP), to replace the frequently used one, Average Precision (AP), for evaluating the performance of video annotation algorithms. General annotation methods just care the number of true-positive samples at the top of the ranked list; they actually do not care the order of these samples. We argue that it is more reasonable to rank "typical" true-positive samples higher than non-typical ones, which can be evaluated by our proposed ATP. However, generally the labels of the training data only differentiate true from false; that is to say, typical or non-typical training samples have the same contribution to the learning process. Therefore, the labels of the unlabeled data learned from these training data can not well measure the typicality. In this paper, we relax the labels of the training data to real-valued typicality scores by a pre-processing stage, which is accomplished by three approaches, including density estimation, user feedback and active learning. Then the typicality scores of the training data are propagated to unlabeled data using manifold-ranking. Experiments conducted on the TRECVID data set demonstrate that this typicality ranking scheme is more consistent with human perception than normal accuracy based ranking schemes.