Generating virtual environments for human communications: Virtual metamorphosis system and novel view generation

At ATR Media Integration & Communications Research Laboratories, our research group is aiming at realizing virtual communication environments in which the users at remote sites can feel that they are virtually co-located. This paper presents the ongoing research in two subareas chosen to address this problem: (a) Generating human images in virtual scenes. (b) An image based approach for generation of (background) scenes. In the virtual metamorphosis system, 3D models of characters are created in advance. For metamorphosis, the facial expressions and body posture of a person are detected in real-time from the face images and monocular thermal images of the person, respectively. The detected expressions and body movements are reproduced by deforming the 3D models. Demonstrations using 3D models of a Kabuki actor and a dinosaur show good performance. For novel view generation, we use as affine coordinate based reprojection scheme, which is also extended to merging real and synthetic objects.