Towards AI-based Semantic Multimedia Indexing and Retrieval for Social Media on Smartphones

To cope with the vastly growing number of Multimedia Assets on Smartphones and Social Media, an integrated approach for Semantic Indexing and Retrieval is required. This paper introduces an approach towards a generic framework to fuse existing image and video analysis tools and algorithms into a Unified Semantic Annotation, Indexing and Retrieval Model resulting in a Multimedia Feature Vector Graph representing various levels of Media Content, Media Structures and Media Features. Utilizing Artificial Intelligence and Machine Learning, these Feature Representations can be used to provide accurate Semantic Indexing and Retrieval. This paper provides an overview of the Generic Multimedia Analysis Framework (GMAF) as well as the definition of a Multimedia Feature Vector Graph Framework (MMFVGF). As a third contribution we introduce AI4MMACCESS to detect differences, enhance Semantics and refine weights in the Feature Vector Graph. Combining this, we describe a solution for highly flexible Semantic Indexing and Retrieval that offers unseen possibilities for applications like Social Media or local Apps on Smartphones.