Image Caption Generation using Vision Transformer and GPT Architecture

Transformer-based models have reshaped image captioning but grapple with issues like caption accuracy, particularly for complex visuals. Addressing these shortcomings is essential. Motivated by existing challenges, the Vision Transformer (ViT) as encoder and Generative Pretrained Transformer 2 (GPT-2) as decoder have been employed to enhance caption quality, utilizing the Seq2Seq framework and training on Flickr8k. This work introduced a novel ViT-GPT-2 image captioning model, evaluating it against benchmarks including Flickr8k. The model excels with BLEU-4 at 39.76 and METEOR at 52.30, bridging visual-textual gaps effectively. This research advances image captioning, offering practitioners an improved model for content indexing, accessibility, and human-computer interaction. ViT- GPT-2’s success underscores bridging semantic gaps in image captioning, with future work exploring diverse datasets and fine-tuning techniques for enhanced performance.