Beyond Raw Videos: Understanding Edited Videos with Large Multimodal Model

The emerging video LMMs (Large Multimodal Models) have achieved significant performance on generic video understanding in the form of VQA (Visual Question Answering), which mainly focuses on raw videos captured with cameras. However, a large portion of videos in real-world applications are edited videos, e.g., users usually cut and add effects/modifications to the raw video before publishing it on social media platforms. The edited videos usually have high view counts but they are not covered in existing benchmarks of video LMMs, i.e., ActivityNet-QA or VideoChatGPT benchmark. In this paper, we take advantage of edited videos on a popular short video platform, i.e., TikTok, and build a video VQA benchmark (named EditVid-QA) covering four typical editing categories, i.e., effect, funny, meme, and game. Funny and meme videos benchmark nuanced understanding and high-level reasoning, while effect and game evaluate the understanding capability of artificial design. Most of the open-source video LMMs perform poorly on the EditVid-QA benchmark, indicating a huge domain gap between edited short videos on social media and regular raw videos. To improve the generalization ability of LMMs, we collect a training set for the proposed benchmark based on both Panda70M/WebVid raw videos and small-scale TikTok/CapCut edited videos, which boosts the performance on the proposed EditVid-QA benchmark, indicating the effectiveness of highquality training data. We also identified the issue in existing video evaluation protocol using GPT-3.5 judge, namely a “sorry” attack, where a sorry-style naive answer can achieve an extremely high rating from GPT judge, e.g., over 4.3 for correctness score on VideoChatGPT evaluation protocol. To avoid the “sorry” attacks, we evaluate results with GPT4 judge and keyword filtering. The dataset is released at https://github.com/xenonlamb/editvid-qa.