Multiple media cues for MPEG-7

This work presents a methodology to extract and represent the semantic content of audio-visual documents. A collection of diverse tools is used to extract low level, signal based descriptions. Joint audio and visual analysis is utilized to automatically extract higher level semantic features. High-level, hand-annotated, descriptors are also used. The hand annotated descriptors are used for retrieval purpose as well as to enhance the results of the automatic procedure, i.e. to allow the system to learn how high level semantic information are linked to low level automatically extracted features through user's input. We draw upon MPEG-7's collection of Descriptors to provide some targets for our audio and visual analysis methods. Selected MPEG-7 Description Schemes, such as the textual description, the description of persons, and the description of the structural aspects of the content of the AV document [1], provide some of the larger containment structures for our features.