Supporting distributed processing of time-based media streams

There are many challenges in devising solutions for online content processing of live networked multimedia sessions. These include content analysis under uncertainty (evidence of content are missed or hallucinated), the computational complexity of feature extraction and object recognition, and the massive amount of data to be analyzed under real-time requirements. In this paper we focus on middleware supporting online media content analysis. Our middleware supports processing, logically organized as a hierarchy of refined events extracted in real time from a set of potentially related time-based media streams. The processing can physically be distributed and redistributed during run time, as a set of interacting components, each performing some content analysis algorithm. The middleware is designed with reuseability, scalability, performance, resource management, and fault tolerance in mind by providing support for mechanisms such as, adaptation, reconfiguration, migration, and replication. The goal is to support applications in trading off the reliability and latency of the content analysis against the available computing resources.