Animated interactive fiction: Storytelling by a conversational virtual actor

This paper presents an architecture for interactive storytelling using state-of-the-art-technology in natural language processing, speech synthesis and 3D-character animation. A conversational 3D-character is used to tell nonlinear stories interactively. Our research focuses on multimodal conversation that combines verbal-vocal with nonverbal-nonvocal natural language. We present a framework for synchronizing speech with facial movements, gesture and body posture by combining findings from linguistics and psychology. The interaction process incorporates the parsing of a textual natural language utterance, the emotion-based triggering of a textual response and its vocal and visual performance-speech synthesis linked with lip-synchronous generation of facial expressions and gestures.