A multi-modal teaching-advisory system using complementary operator and sensor information

A multimodal teaching-advisory system for sensor-based robot systems is presented that uses complementary information obtained from the operator and sensors. We propose a new framework for on-site robot teaching systems that takes account of the characteristics of humans and sensors. This framework enables the teaching system to integrate a variety of robot task constraints, task specifications, their tolerances as defined by the operator, and sensing information acquired by the sensors in a complementary manner. The system synthesizes all of this information into a form that is easily understood by the operator. Finally, the synthesized information is presented to the operator as multimodal information. By combining this advisory information with the operator's own information, the desired robot teaching commands can be generated. This system can significantly improve total robot system performance by ensuring high-quality teaching.