Cross-Linguistic Study on Affective Impression and Language for Visual Art Using Neural Speaker

Visual art is one of the ideal targets for affective computing because viewing artworks is a regular experience for many people, and it elicits various affective appraisals, cognitions, and reactions in the viewer. To gain a detailed understanding of the interplay between visual content, its emotional impact, and linguistic explanations of this impact, visual art datasets have been proposed. One example is the ArtEmis dataset, which contains emotion categorization and linguistic expressions by crowds of people in reaction to numerous paintings. Linguistic expressions are influenced both by culture in how to appraise art and by language in how to verbalize one's impression. However, cultural and linguistic differences in this domain have not been fully explored. Therefore, we collected a new dataset (ArtEmis-JP) consisting of 16 k emotion labels and utterances in Japan, one of the countries most frequently compared with Western countries in psychology, while carefully following the procedures of the original ArtEmis study conducted with English speakers. In this paper, we report the commonalities and differences between the original ArtEmis dataset and our Japanese dataset using basic statistical comparisons and performance comparisons for a state-of-the-art neural speaker in utterance generation. Going beyond the original study, we newly examined the impact of expertise on emotional categorization and linguistic expressions by examining the differences between experts and non-experts.