Multi-modal Emotion Recognition Utilizing Korean-English Vision and Language Information Alignment Pre-trained Model

Emotions in humans find expression through a variety of modalities. In this paper, we build a new dataset that combines existing vision and language datasets and use this newly formed multi-modal dataset for the task of multi-modal emotion recognition in empathetic conversation. We utilize a vision-language pre-trained model, VL-KE-T5, to build a model that can process image and text information simultaneously. Comparative experiments show that the proposed model outperforms models that handle image and text separately in emotion recognition.