Multi-Head Attention on Image Captioning Model with Bert Embedding

Image captioning models are in abundance, with various techniques and methods in place for them. One such primitive model is "Show, Attend and Tell". This model showcases how well attention mechanisms can be used to focus on necessary parts of an image to predict the next word in the caption. While this model uses soft attention mechanism, the recent development of transformer introduced the concept of multi-head attention mechanism, which was the inspiration for using it instead of soft attention. Multi-head attention was primarily introduced to work on text, but we experimented it on images and text here. We show different components of the captioning model such as the encoder (ResNet), decoder (LSTM with Bert Embeddings) and the use of multi-head attention for image and text instead of text only (as previously used in transformer). We work on the benchmark dataset of MSCOCO. We prepared a baseline and achieved better BLEU score results with our proposed model.