Analysis of visual similarity in news videos with robust and memory-efficient image retrieval

Many large collections of news videos dating back several decades can now be accessed online. For users to easily retrieve a compilation of stories on a particular event/topic and to quickly sample each story clip, all the news videos must be precisely segmented into stories and a representative video summary must be generated for each story. In this paper, we demonstrate that effectively exploiting the visual similarities pervasive in all news videos can greatly help to fulfill these technical requirements and thus enable the dynamic retrieval and mixing of small news video fragments. Two new algorithms are developed to accurately detect two important sources of visual similarity: (1) similar preview and story frames, and (2) repeated appearances of a news anchor. As a result, valuable sources of preview clips and informative clues about story boundaries are obtained from identification of these visual similarities. The retrieval engine implemented in both algorithms employs compact global image signatures and requires a small memory footprint, so that many instances of the detection algorithms can run concurrently on the same server for fast processing of a large collection of news videos. At the same time, the retrieval engine is robust to the large appearance variations encountered in the preview matching and anchor detection problems. In addition, since the video frame's color information is not required in our algorithms, both modern color and vintage black-and-white news footage can be processed in the same framework.