A Cross-Modal Spatio-Temporal Interaction Network for Video Question Answering

Video Question Answering (VideoQA) requires models to comprehend video content and generate answers to natural language questions. VideoQA must reason over both spatial and temporal dimensions, presenting unique challenges as questions require varying degrees of spatial and temporal visual information. This paper proposes a Cross-Modal Spatio-Temporal Interaction Network that adaptively performs spatial and temporal interactions between video and text modalities based on question intent, without requiring additional annotations. Our approach integrates feature representation, intra-modal perception, cross-modal spatio-temporal interaction, and answer generation. The model extracts video and question features, introduces learnable tokens for global semantics and spatio-temporal intent, and employs attention mechanisms to adaptively fuse spatial and temporal information. Experiments on MSVD-QA and MSRVTT-QA datasets demonstrate that our method achieves competitive performance, with achieving $48.4 \%$ accuracy on MSVD-QA, outperforming the second-best method by 2.4%. Ablation studies verify the effectiveness of each proposed module, with visualizations confirming the model’s ability to adaptively focus on spatial or temporal information based on question intent.