MOFNet: Memory Optimization Fusion Network for Query-Focused Video Summarization

Video summarization aims at selecting key frames or shots with important information from a long video to create a shorter video that captures the essence of the original video content. Query-focused video summarization (QFVS) is a key research focus as it incorporates user subjectivity into the summarization process, further enhancing the user experience. Visual features extraction and multimodal features fusion are two key technologies for QFVS, directly affecting the accuracy of the summarization results. However, existing methods neither fully consider the redundancy in extracted visual features nor pay sufficient attention to enhancing the expressive power of multimodal fusion features. To address these problems, in this article, we propose a novel memory optimization fusion network (MOFNet), which focuses on reducing the redundancy of visual features at the extraction phase and exploring the temporal correlation information of visual features during the multimodal feature fusion phase. Specifically, our MOFNet mainly consists of a global memory optimization module (GMOM) and a cross-modal fusion augmented module (CFAM). First, in the GMOM, we employ a multihop operation based on the memory network to optimize the extracted visual features to obtain more useful global visual information. Second, in the CFAM, we extract the temporal sequence information and fuse the visual features and query concepts, enhancing the expressive ability of multimodal features further. Finally, we calculate the shot-level importance score by the fused features after augmented processing, so as to select the shots with high scores to form a video summary. Extensive experiments on two datasets prove the effectiveness of our proposed method.