Continuous estimation of emotional change usingmultimodal affective responses

Emotions have a significant effect on our daily behavior, such as perception, memory, and decision making. For this reason, interest in considering the emotions of the user in a human-computer interface has recently increased. This is important for future interface applications, which are expected to operate in harmony with humans. In this paper, we present our approach to instantaneously detecting the emotions of video viewers from remote measurement using an RGB camera. Facial expression and physiological responses, such as heart rate and pupil diameter, were measured by analyzing facial videos. We also verified the effectiveness of the contactless measurement by acquiring electroencephalogram signals using a contact-type electroencephalograph. By combining the measured responses into multimodal features and using machine learning, we showed that the results of emotion estimation were better than estimates made from only single-mode features.