VR Interaction in Real-Time between Avatar with Voice and Gesture Recognition System

In this research, a dialog environment between human and virtual agent has been constructed. With the commercial off-the-shelf VR technologies, special devices such as data glove have to be used for the interaction. But this is difficult to manipulate objects. If there is a helper who has direct access to objects in a virtual space, we may ask him. The question, however, is how to communicate with the helper. This paper presents a solution to the question. The basic idea is to utilize speech recognition and gesture recognition systems. Experimental results have proved the effectiveness of the approach in terms of facilitating man-machine interaction and communication. The environment constructed in this research allows a user to communicate by talking and showing gestures to a personified agent in virtual environment. A user can use his/her finger to point at a virtual object and ask the agent to manipulate the virtual object.