Combining Emotional Gestures, Sound Effects, and Background Music for Robotic Storytelling - Effects on Storytelling Experience, Emotion Induction, and Robot Perception

Storytelling is a long-established human tradition for entertainment and knowledge transfer. Social robots are emerging as a new storytelling medium, being able to imitate human storytelling using gestures but also extend it by adding, e.g., sound effects to the experience. Due to COVID-19 restrictions, we conducted an online video-based study to investigate the effects of congruent respectively incongruent or no gesture usage in combination with additional non-speech sounds, i.e. sound effects and background music, on recipients’ transportation into the story told, emotion induction, and perception of the robot. Results indicate no effect of additional non-speech sound integration on the variables listed above. Contradicting with related findings from in-person studies, we found a no significant differences between congruent, incongruent and no gesture usage. Last, no interplay of additional sounds and gesture congruence was identified. Future studies should provide deeper insights into the importance of multimodal congruence in video-taped robots and the possible advantages of adding non-speech sounds to online but also in-person robotic storytelling as well as their interplay in in-person HRI.CCS CONCEPTS • Computer systems organization → Robotics; • Human-centered computing → Empirical studies in HCI.