Semantic Indexing of News Video Sequences: A Multimodal Hierarchical Approach Based on Hidden Markov Model

In this paper, we propose a system that semantically classifies news video at different layers of semantic significance, using different elements of visual content. The classification hierarchy generates low-level concepts, and concept hierarchy generates high-level concepts from low-level concepts. Our classification hierarchy is based on a few popular audio and video content analysis techniques like short-time energy and zero crossing rate (ZCR) for audio and hidden Markov model (HMM) based techniques for video, using color and motion as features. From these low-level concepts of classification hierarchy, rule-based classifier generates higher-level concepts. The proposed framework has been successfully tested with TV news video sequences. Our results are very promising. We have used top-down hierarchical approach, which permits us to avoid shot detection and clustering and consequently improves the classification performance. This has created the great opportunity for supporting more powerful video search engines.