Scheduled System Maintenance:
On Monday, April 27th, IEEE Xplore will undergo scheduled maintenance from 1:00 PM - 3:00 PM ET (17:00 - 19:00 UTC). No interruption in service is anticipated.
By Topic

Toward multimodal interpretation in a natural speech/gesture interface

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)
Kettebekov, S. ; Dept. of Comput. Sci. & Eng., Pennsylvania State Univ., University Park, PA, USA ; Sharma, R.

Hand gestures and speech comprise the most important modalities of human to human interaction. Motivated by this, there has been a considerable interest in incorporating these modalities for “natural” human-computer interaction (HCI) particularly within virtual environments. An important feature of such a natural interface would be an absence of predefined speech and gesture commands. The resulting bimodal speech/gesture HCI “language” would thus have to be interpreted by the computer. This involves challenge ranging from the low-level signal processing of bimodal (audio/video) input to the high level interpretation of natural speech/gesture in HCI. This paper identifies the issues of natural (non-prefixed) multimodal HCI interpretation. Since, in the natural interaction, gestures do not exhibit one-to-one mapping of their form to meaning, we specifically address problems associated with vision-based gesture interpretation in a multimodal interface. We consider the design of a speech/gesture interface in the context of a set of spatial tasks defined on a computerized campus map. The task context makes it possible to study the critical components of the multimodal interpretation and integration problem

Published in:

Information Intelligence and Systems, 1999. Proceedings. 1999 International Conference on

Date of Conference:

1999