E-VChat: A video communication system in which a speech-driven embodied entrainment character working with head motion is superimposed for a virtual face-to-face scene

We developed an embodied video communication system in which a human-type avatar called “VirtualActor,” engaging in interactive behavior, was superimposed on the other speech partner's video image in a virtual face-to-face scene. The system posed some problems for VirtualActor such as a sense of restraint owing to sensor cables and the sensors' lack of portability. In this paper, we propose the concept of an embodied video communication system in which a CG character is superimposed on the other talker's video image in a face-to-face scenario, and we develop a prototype called “E-VChat,”. This system has a CG character, which moves on the basis of the talker's head motions with a motion-capture device and automatically generates motions on the basis of the talker's voice. Furthermore, we propose a multiple-character system in which CG characters crowd around the other talker virtually, and we develop the prototype system. These characters enhance human communication by facilitating interactive behavior on the basis of the talker's voice. An interview-style communication experiment is performed to confirm the effectiveness of the E-VChat system for 12 pairs of subjects using three communication modes: “only the other talker's image,” “the other talker's image with the talker's avatar,” and “the other talker's image with the talker's avatar and character audience.” The results show that all communication modes are positively assessed by sensory evaluation and that the other talker's image with talker's avatar and character audience mode is evaluated highly in paired comparison.