Integrating natural language and gesture in a robotics domain

For our work in the robotics domain, we have concentrated on integrating spoken natural language and natural gesture for command and control of a semi-autonomous mobile robot. We have assumed that both spoken natural language and natural gesture are more user-friendly means of interacting with a mobile robot, and from the human standpoint, such interactions are easier, given that the human is not required to learn additional interactions, but can rely on "natural" ways of communication. So-called "synthetic" methods, such as data gloves, require additional learning; however, this is not the case with natural language and natural gesture. We, therefore, rely on what is natural to both spoken language when it is used in conjunction with natural gestures for giving commands. Furthermore, we have been integrating these interactions with the robotics components as the robotics system is being developed. The interface is not treated as an ad hoc add-on or patch. By doing so, we believe the interface will be more robust and because it is being integrated during system development, we hope to achieve a more seamless interface, one which both acts and feels as an integral part of the robotics application. We discuss the kinds of interactions which our system is currently capable of performing. We also discuss the processing of the various input to produce an appropriate robotic response, and finally, we discuss what future kinds of interactions we would like to incorporate into the system, and what will be required to achieve these results.