Combining Emotional Gestures, Sound Effects, and Background Music for Robotic Storytelling - Effects on Storytelling Experience, Emotion Induction, and Robot Perception

Storytelling is a long-established human tradition for entertainment and knowledge transfer. Social robots are emerging as a new storytelling medium, being able to imitate human storytelling using gestures but also extend it by adding, e.g., sound effects to the experience. Due to COVID-19 restrictions, we conducted an online video-based study to investigate the effects of congruent respectively incongruent or no gesture usage in combination with additional non-speech sounds, i.e. sound effects and background music, on recipients’ transportation into the story told, emotion induction, and perception of the robot. Results indicate no effect of additional non-speech sound integration on the variables listed above. Contradicting with related findings from in-person studies, we found a no significant differences between congruent, incongruent and no gesture usage. Last, no interplay of additional sounds and gesture congruence was identified. Future studies should provide deeper insights into the importance of multimodal congruence in video-taped robots and the possible advantages of adding non-speech sounds to online but also in-person robotic storytelling as well as their interplay in in-person HRI.CCS CONCEPTS • Computer systems organization → Robotics; • Human-centered computing → Empirical studies in HCI.

1 INTRODUCTION

Storytelling is almost as old as mankind’s history itself, as it has always been used to pass on knowledge to subsequent generations [41]. Traditional storytelling is characterized by a "single teller addressing an audience using speech, physical gestures, possibly props, and non-speech sounds." [86, p. 1] As the foundation of human knowledge structures and memory, stories are an integral part of social interaction [85]. In terms of human-human communication, the above described multimodality plays an important role, as it benefits the speaker at various levels. Not only can it help to communicate more efficiently by providing complementary information. Also, providing redundant information by using different modalities simultaneously supports robustness. Thus, information provided in a specific modality can be used to enhance, disambiguate, or highlight information transported via another modality [81]. Consistency is crucial here. Walker-Andrews et al., for ex, have observed that babies can only recognize emotional facial expressions when they are accompanied by appropriate vocal expressions [82], [84]. Consistency, i.e. corresponding verbal and non-verbal behaviour, also leads to a higher level of perceived likability in adults [84]. Similarly, human-robot interaction (HRI) benefits from multimodality (for a survey see [48]). To this end, the importance of consistency is also indicated in HRI, for example between vocal expressions and body postures [77] or facial expressions [4].