Visual-Aware Attention Dual-Stream Decoder for Video Captioning

Video captioning is a challenging task that captures different visual parts and describes them in sentences, for it requires visual and linguistic coherence. The attention mechanism in the current video captioning method learns to assign weight to each frame, promoting the decoder dynamically. This may not explicitly model the correlation and the temporal coherence of the visual features extracted in the sequence frames. To generate semantically coherent sentences, we propose a new Visual-aware Attention (VA) model, which concatenates dynamic changes of temporal sequence frames with the words at the previous moment, as the input of attention mechanism to extract sequence features. In addition, the prevalent approaches widely use the Teacher-forcing (TF) learning during training, where the next token is generated conditioned on the previous ground-truth tokens. The semantic information in the previously generated tokens is lost. Therefore, we design a Self-forcing (SF) stream that takes the semantic information in the probability distribution of the previous token as input to enhance the current token. The Dual-stream Decoder (DD) architecture unifies the TF and SF streams, generating sen-tences to promote the annotated captioning for both streams. Meanwhile, with the Dual-stream Decoder utilized, the ex-posure bias problem is alleviated, caused by the discrepancy between the training and testing in the TF learning. The effectiveness of the proposed Visual-aware Attention Dual-stream Decoder (VADD) is demonstrated through the result of ex-perimental studies on Microsoft video description (MSVD) corpus and MSR-Video to text (MSR-VTT) datasets.