Interaction with virtual environment using verbal/non-verbal communication

Verbal/non-verbal communication between a human and a computer system in a virtual space is proposed. As a concrete example, we have applied it to the field of mechanical assembly domain. The system accepts questions/commands from a user in a spoken language and is successful in the correct interpretation of the user's intentions and in the maintenance of the communication between the user and the system. Using non-verbal operations such as pointing actions, a user is able to point out an object simply. Because there is no depth feeling in the virtual space used in the system, it is often difficult for a user to grasp or point to an object using a data glove. The way a user instructs a system to assemble a virtual machine is reported in this paper. We have reported the way that the system watches a user's behavior while he/she is assembling a virtual machine, but the system simply points at the part being erroneously operated; no instruction is given to the user. An avatar should perform this instruction along with a vocal explanation for manipulating mechanical parts. In the same way, for questions from a user, the avatar should answer with a gesture and in a spoken language. If this bi-directional verbal/non-verbal communication can be realized, a user has only to imitate the avatar's action. This means that mutual comprehension will be promoted.