Advancing Video Summarization Using Language-Based Attention Transformer

Video summarization is a process geared towards condensing lengthy videos into concise summaries, facilitating efficient browsing and context understanding. Traditionally, supervised learning-based approaches have been introduced to capture significant content within video, and steer the summarization task by leveraging continuous frame-level data, though they suffer from semantic gaps between what is depicted visually and what is contextually meaningful, offer limited generality and limited generalization, and rely heavily on human-curated summaries for training. In response to these gaps, a transformer- based framework is proposed, as a novel direction, primarily built on a Cross-Attention Transformer (CAT), which uses CLIP ViT-B/32 for visual or frame features and Clip for caption features. Further, we introduced a language-guided multimodal transformer to twofold support, firstly to assess the importance of each video relative to one another in an automatically generated and comprehensive video caption, and secondly, to compare the similarity score of each frame with caption tokens to get the gist of the video. Through extensive evaluations of standard video summarization datasets (TVSum and SumMe) using F1 score and BLEU metrics, the proposed framework demonstrates significant performance improvements of about 3% over baseline methods and prior approaches. The experimental evaluations exhibit substantial advancements in transfer learning scenarios, showcasing its robust generalization capabilities and being adapted to a zero-shot learning setting by training without ground-truth supervision.Our repository of codes is available at this link: https://github.com/hkhitesh7/Video-Summarization-using-CAT.