User interaction in mixed reality interactive storytelling

In this paper, we describe a mixed reality system based on a "magic mirror" model, in which the user's image is captured in real time by a video camera, extracted from his/her background and mixed with a 3D graphic model of a virtual image including the synthetic characters taking part in the story. The resulting image is projected on a large screen facing the user, who sees his/her own image embedded in the virtual stage with the synthetic actors. The graphic component of the mixed reality world is based on a game engine, Unreal Tournament 2003. This engine not only performs graphic rendering and character animation but incorporates a new version of our previously described storytelling engine. A single 2D camera facing the user analyses the image in real-time by segmenting the user's contours.