Leveraging the Video-Level Semantic Consistency of Event for Audio-Visual Event Localization

Audio-visual event (AVE) localization has attracted much attention in recent years. Most existing methods are often limited to independently encoding and classifying each video segment separated from the full video (which can be regarded as the segment-level representations of events). However, they ignore the semantic consistency of the event within the same full video (which can be considered as the video-level representations of events). In contrast to existing methods, we propose a novel video-level semantic consistency guidance network for the AVE localization task. Specifically, we propose an event semantic consistency modeling (ESCM) module to explore video-level semantic information for semantic consistency modeling. It consists of two components: a cross-modal event representation extractor (CERE) and an intra-modal semantic consistency enhancer (ISCE). CERE is proposed to obtain the event semantic information at the video level. Furthermore, ISCE takes video-level event semantics as prior knowledge to guide the model to focus on the semantic continuity of an event within each modality. Moreover, we propose a new negative pair filter loss to encourage the network to filter out the irrelevant segment pairs and a new smooth loss to further increase the gap between different categories of events in the weakly-supervised setting. We perform extensive experiments on the public AVE dataset and outperform the state-of-the-art methods in both fully- and weakly-supervised settings, thus verifying the effectiveness of our method.

I. Introduction

Research on the audio-visual event (AVE) localization task has proven that joint audio-visual representations can facilitate understanding unconstrained videos [1], [2], [3], [4], [5], [6]. Specifically, AVE is defined as an event that is audible and visible simultaneously (as shown in Fig. 1). AVE localization requires the model to identify the AVE category and localize its boundary in the temporal dimension. It is essential for understanding, recommending, and searching video content, especially for short video platforms.