Automated Image Caption Generation Framework using Adaptive Attention and Bi-LSTM

Many Attention-based deep architectures have been widely adopted for image captioning. Mostly force visual attention to be active for word generation. However, spatial region specific word generation is not desired always. In this paper, to address this issue, an adaptive attention mechanism-based automated Image caption generation framework is proposed. Our model uses Inception-V3 to extract various global spatial features and the adaptive attention module helps to decide whether to attend to the image (and if so, to which regions) or to the visual sentinel maps. Further, at the decoding end, a Bi-LSTM network refines the text description. The experimental results, usingFlickr8K and Visual Assistance Datasets, demonstrate that the proposed model provides significant improvements over similar state-of-the-arts in terms of BLEU scores.