Multimodal acting in mixed reality interactive storytelling

In this paper, an experimental mixed reality using a multimodal approach is introduced which lets users play characters in interactive narratives as though acting on a stage. Users interact with characters through speech, attitude, and gesture, enhancing their immersion in the virtual world. This system provides a small-scale but complete integration of multimodal communication in interactive storytelling. It uses a narrative's semantic context to focus multimodal input processing-that is, the system interprets users' acting (the multimodal input) in the mixed reality stage in terms of narrative functions representing users' contributions to the unfolding plot.