Hierarchically Multi-modal Indexing of Sports Video

A multi-level structure of video contents is introduced and a novel mechanism of hierarchically indexing sports video using multi-modal cues is proposed. Firstly, video is indexed with information from video and audio individually. Video is segmented into physical shots based on visual features, and then identified as syntactic shots on the middle level features according to broadcasting characteristics. Audio is analyzed to get physical and syntactic contents step by step. Then, fusing the informative clues from video, audio and text extracted from video, semantics of sports video can be generated and shots are annotated as semantic shots. Considering the difficulty of extracting full semantics of video, a multi-modal retrieval is proposed to compensate query by event