Multimodal Video Summarization using Attention based Transformers (MVSAT)

With the sharp increase in consumption and creation of content in the format of videos it has become a necessity to have mechanisms to analyze and extract most vital information delivered in a shorter format. This research tends to delve deeper into enhancing video summarization using attention-based transformers by embedding present state-of-the-art models in the feature extraction phases. This paper demonstrates the effective utilization of video frames to extract visual features and also generate captions that serve as the second source of modality. These captions tend to overcome the absence of transcripts and audio portions needed for summarization and at the same time present an effective way to extract textual features. The extracted features are then handled by an effective transformer model to select the keyframes needed for summarization. The resultant methodology delivers a better F1 score compared to other supervised and unsupervised models and the mechanisms considering a single modality. The method presented also goes on to provide a full-fledged summarized video generated from the selected keyframes thus making it an end-to-end process for video summarization. SumMe and TVSum datasets have been extensively used for training and testing purposes throughout this research and the proposed model has achieved an F1 score of 54.91 on SumMe and 63.34 on TVSum dataset.