A novel error measure for the evaluation of video indexing systems

Error evaluation of video indexing systems is a problem for which no satisfactory solution has been presented so far. This paper introduces a new error measure for evaluating the results of video indexing systems. The measure compares the scene and shot boundaries of the correct index of a video sequence with the automatically created index, and decides dependent on predefined penalties whether a scene boundary is falsely detected, undetected or correctly detected. Based on this decision the measure calculates the error rates for the segmentation capability of the indexing system. Then the algorithm extends the reference and the test index by the falsely detected and undetected boundaries and compares the content classes of the resulting indexes to determine the classification error rate. The presented measure is compared to two existing measures, and the pros and cons of each measure are listed.