Video abstraction: summarizing video content for retrieval and visualization

We describe a scheme for summarizing video content using mosaics and moving object trajectories, and an approach exploiting this scheme for content-based retrieval and scene visualization. Video is segmented into clips using scene cut detection. Within each clip, static scene content is summarized by a mosaic, and dynamic content, by segmenting and tracking moving objects. If available, sensor view point data is used to map these summaries to physical coordinates. A collection of clips can be browsed or searched on appearance as depicted by the mosaic, or on activity as depicted by tracks. The mosaic and tracks support an enhanced video viewer that allows the contents of entire clips to be examined rapidly-for example, as tracks overlayed upon a mosaic.