Bridging Between Emojis and Kaomojis by Learning Their Representations from Linguistic and Visual Information

Small images of emojis have unique characteristics as additional information in understanding writers' intentions. They enable social media users to emphasize their emotions and to express gestural movements in their posts. In addition to emojis, kaomojis (emoticons or facemarks) also behave in a similar way. They are composed of a sequence of characters, which are popularized especially in Asian countries. Although both emojis and kaomojis fulfill similar functions and share the same meaning that can be clues in opinion mining or sentiment analysis, the previous researches have been biased to explore emojis and kaomojis separately. In this paper, we align emojis and kaomojis together as a single token in the Japanese context to offer a bridge between them. Specifically, we aim to judge whether emojis and kaomojis share the same meaning or are similar with each other. We assume that emojis and kaomojis are both a single word in order to obtain their linguistic information with the skip-gram model. Furthermore, we present a new approach to consider the appearances of emojis and kaomojis in themselves, meaning that we explore the information of their visually similar shapes. We regard both of them as a single image to take into account their visual information with the CNN model. We merge two different perspectives toward emojis and kaomojis by exploring their linguistic and visual information simultaneously on the same space. The experimental results showed that we can align an unlimited number of emojis and kaomojis together with their representations (embeddings), and adding the visual information to the linguistic information can improve their representations.