Improving Video Retrieval Systems Using Multimodal Fusion and Re-Ranking Algorithm

Under the strong development of multimodal models, the application of these models in video or image retrieval systems is gradually becoming a trend. Researchers are striving to build or utilize models capable of effectively representing features of images and videos. However, constructing multimodal models often incurs significant training costs, especially for datasets containing large volumes of video and image data. In this paper, we propose a method that combines features from different multi-modal models which specifically are BLIP and InternVideo and this method does not cost any training costs but only takes advantage of existing multi-modal model in video retrieval systems. By combining both types of features, we aim to enrich the diversity and comprehensiveness of video features that will improve the performance of the video retrieval system. Additionally, we propose a re-ranking algorithm based on information from audio and text appearing in the video which are aspects that multi-modal models typically struggle to process. We conducted experiments on the MSR-VTT dataset to compare our method with results obtained using individual models. Furthermore, we also tested on the Vietnam Media News Dataset from the HCM AI Challenge 2022 to validate the retrieval accuracy of our system when combined with our re-ranking algorithm. Experimental results have demonstrated the effectiveness and accuracy of our approach.)