Current Multimedia Data Formats and Semantic Computing: A Practical Example and the Challenges for the Future

Research is doing exciting progress in understanding images as more than just a set of pixels, videos as more than just an even greater set of pixels synchronized with time, or audio streams as more than just an array of samples. Current implementations of multimedia data formats, such as MPEG, however, still focus on the storage of pixels and samples. This article uses a straightforward example of a system where a pixel-based video stream is superimposed on a vector-based animation to show that the lack of capability to store higher-level semantic information leaves the developers only two choices: Either accept suboptimal quality or design a new non-standard format. The paper presents an in-depth discussion of a practical example and draws conclusions for the future directions of multimedia data formats.