Increasing Realism and Supporting Content Planning for Dynamic Scenes in a Mixed Reality System incorporating a Time-of-Flight Camera

Mixed reality is the combination of real and virtual scene content. Besides correct alignment of the two modalities and correct occlusion handling the core issues to be tackled are the degree of realism and the ease of use. For a convincing perception correct occlusion handling and shadowing is mandatory. We present a system for mixed reality applications which features enhanced realism through automatic shadow computation and supported placement functions for virtual content. The system relies on the incorporation of a ToF-camera device. This device delivers real-time depth maps of the environment and is used to build a static environment model. It also allows correct handling of mutual occlusion between real and virtual content, shadow computation and enhanced content planning. Other mixed reality systems require expensive studio setups with complex camera tracking installations. Often multi- camera approaches in combination with chroma-keying facilities are used. The presented system is rather inexpensive, compact, mobile, flexible and provides convenient calibration procedures. The standard chroma-keying procedure is replaced by the usage of the environment model. The planning and alignment of virtual content is simplified by automatic extraction of dynamic content in the scene. This is essential for the placement of virtual content. With the presented system appropriate depth maps of the mixed content are available in real-time, which makes the approach suitable for 3DTV productions. The presented paper discusses all key elements of mixed-reality applications based on the presented approach. This includes camera calibration, environment model generation, real-time handling between interacting virtual and real content, shadowing for virtual content and dynamic object tracking for content planning.