Integration of gaze and gesture detection in nature language instructing of robot in an assembly scenario

We present the development of and experiments with a robot system showing multimodal interaction capabilities. We focus on the understanding of human instructions in natural language by integrating gaze and pointing hand gestures. A typical application of such a system is interactive assembly. A human communicator sharing a view of the assembly scenario with the robot instructs the latter by speaking to it in the same way that he would communicate with a child. His instructions can be under-specified, incomplete and/or context-dependent. By detecting gaze and gesture information of the instructor, which are described in two sections of the paper, the ambiguities in the interactions can be resolved in a natural manner. Finally, we outline a list of future topics for extending our research.