Studying Effects of Incorporating Automated Affect Perception with Spoken Dialog in Social Robots

Social robots are becoming an integrated part of our daily lives with the goal of understanding humans' social intentions and feelings, a capability which is often referred to as empathy. Despite significant progress towards the development of empathic social agents, current social robots have yet to reach the full emotional and social capabilities. This paper presents our recent effort on incorporating an automated Facial Expression Recognition (FER) system based on deep neural networks into the spoken dialog of a social robot (Ryan) to extend and enrich its capabilities beyond spoken dialog and integrate the user's affect state into the robot's responses. In order to evaluate whether this incorporation can improve social capabilities of Ryan, we conducted a series of Human-Robot-Interaction (HRI) experiments. In these experiments the subjects watched some videos and Ryan engaged them in a conversation driven by user's facial expressions perceived by the robot. We measured the accuracy of the automated FER system on the robot when interacting with different human subjects as well as three social/interactive aspects, namely task engagement, empathy, and likability of the robot. The results of our HRI study indicate that the subjects rated empathy and likability of the affect-aware Ryan significantly higher than non-empathic (the control condition) Ryan. Interestingly, we found that the accuracy of the FER system is not a limiting factor, as subjects rated the affect-aware agent equipped with a low accuracy FER system as empathic and likable as when facial expression was recognized by a human observer.