Exploring Audio-Visual Concepts for Dense Video Captioning

Dense video captioning involves temporally localizing multiple events within a video and describing them using natural language. This paper introduces the Audio-Visual Concept-aware Captioning (AVCC) model, which explores the relationship between audio and visual modalities for generating more accurate captions and localization. First, AVCC takes audio and visual modalities as input and integrates them using audio-guided visual attention to provide more precise temporal information. Additionally, AVCC predicts potential concepts for each video segment and combines them with video features to enrich the captions. Extensive experiments demonstrate that AVCC achieves superior performance compared to competing methods. Moreover, evaluating different input modalities shows that audio-guided visual attention effectively captures temporal boundaries, enhancing event localization. Ablation studies and quantitative analysis further confirm that concept detection identifies key concepts, leading to more detailed captions.