Facial Emotion Recognition in Indonesian Image Captioning using CNN and LSTM

Image captioning, a prominent field in artificial intelligence, aims to provide textual descriptions of images. While extensive research has been conducted in English, there is a dearth of studies in the Indonesian language. Moreover, existing approaches predominantly focus on factual descriptions, neglecting emotional aspects. This study addresses this gap by integrating facial expression features obtained through Facial Emotion Recognition (FER) into Indonesian image captioning using Convolutional Neural Network (CNN) and Long Short-term Memory (LSTM) methods. Leveraging datasets FER2013 and filtered Flickr30K. The results are evaluated by using BLEU (Bilingual Evaluation Understudy), Referring to the BLUE (Bilingual Evaluation Understudy), METEOR (Metric for Evaluation of Translation with Explicit Ordering) and also human evaluation. The result which are evaluated using BLEU-1, BLEU-2, BLEU-3. BLEU-4, and METEOR obtained score 84.96, 79.76, 68.41, 72.35, and 36.33 respectively. The results of human evaluation showed that an average of 67.5 respondents thought that the caption results matched the image.