Semantic highlight retrieval

Finding highlights relevant to a text query in unedited videos has become increasingly important due to their unprecedented growth. We refer this task as semantic highlight retrieval and propose a query-dependent video representation for retrieving a variety of highlights. Our method consist of two parts: (1) “viralets”, a mid-level representation bridging between visual and semantic spaces; (2) a novel Semantically-Modulation (SM) procedure to make viralets query-dependent (referred to as SM viralets). Given SM viralets, we train a single highlight ranker to predict the highlightness of clips with respect to a variety of queries, whereas existing approaches can be applied only in a few predefined domains. We collect a viral video dataset¹ including users' comments, highlights, and/or original videos. Among a database with 1189 (13% highlights) clips, our highlight ranker achieves 41.2% recall at top-10 retrieved clips. It is significantly higher than a state-of-the-art domain-specific highlight ranker and its extension. Similarly, our method also outperforms all baseline methods on the video highlight dataset.