Cross-modal Complementary Learning and Template-based Reasoning Chains for Future Event Prediction in Videos

Although multi-modal large language models (MLLMs) have impressive cross-modal reasoning and prediction capabilities, a unified and rigorous evaluation standard is still lacking. In this paper, we propose a future event prediction task to evaluate their cross-modal temporal prediction capability. This task requires the model to generate descriptions of events that may occur in future based on the input premise video. We build a dataset on the existing datasets for model evaluation. This task faces many challenges, including the complexity of processing video data, such as understanding changes in objects, actions, and time dimensions within the video and the interference of redundant information. To address these challenges, we propose a novel cross-modal prediction framework that introduces cross-modal supplementary learning and template-based reasoning chains based on MLLMs. Cross-modal supplementary learning aims to promote visual and text information to supplement and mine their respective information, primarily to capture critical information in videos, relying on the adaptive temporal filter and casual Q-Former. The template-based reasoning chain drives GPT-4 to generate a series of template question pairs through design prompts, gradually guiding the model to perform hierarchical reasoning to support the final prediction. Through experimental evaluation, the performance of the current MLLMs may not meet the requirements, and our model outperforms all existing models in predicting future events. It shows that the capabilities of MLLMs can be further explored.