GPT-Based Knowledge Guiding Network for Commonsense Video Captioning

Video-based commonsense captioning aims to generate captions for the video content while providing multiple commonsense about the underlying event. Existing methods utilize video features to explore and generate commonsense containing latent semantics. However, this process needs to overcome the complex semantic gap between visible videos and invisible commonsense, which is not supported by the limited knowledge in existing video captioning datasets. To this end, we propose a novel GPT-based Two-stage Knowledge Guiding Network (TKG-Net), which uses GPT to augment datasets knowledge and introduces a cross-attention mechanism to fuse multimodal knowledge. Specifically, to augment knowledge, we set prompts and finetune GPT to imagine and reason based on the video content description at the first stage. At the second stage, to prevent over-reasoning caused by the loss of visual features in GPT, TKG-Net extracts high-level semantic representations of commonsense knowledge and fuses them with video features in a cross-attention mechanism for multimodal semantic interaction. Our experiments on the large-scale Video-to-Commonsense dataset manifest significant improvements over the previous state-of-the-art approach on all metrics.