Image Caption Validation for Public Complaints on Social Media

Image Caption Validation is the task of validating whether a caption matches the image provided. The result of this task can reduce distress by validating complaints conveyed from the local community. In this paper, we introduce a custom data set of Indonesian public complaints that consists of floods and littering complaints from Twitter. We used a fine-tuned VGG16 model for image classification and an IndoBERT Model (Indonesian Bert Model) for text classification. Both of these models were combined to into a system that validates whether a given caption accurately describes the image. The experimental result showed that the fine-tuned VGG16 had an accuracy score of 93%, the IndoBERT Model had an accuracy of 89%, and the Validation Output Prediction Model had an accuracy of 56%.