Development of a Synchronized Human-Robot-Virtuality Interaction System using Cooperative Robot and Motion Capture Device

Teleoperation robot systems have been developed to assist and even replace human in the fields of service, space exploration and medical healthcare. In particular, human-robot interaction stimulates the development of teleoperation robot system, hence, intuitive and effective control interaction remains a significant component within teleoperation system.The robot needs to reflect the real intention of the human during operation, therefore the differences between human and robot requires to be considered appropriately. In this study, we proposed a synchronized human-robot-virtuality interaction system, in which the physical interaction and the virtual interaction are integrated to achieve an intuitive and immersive interaction. The motion data are acquired and transmitted to the virtual environment and the cooperative robot using the motion capture system. The virtual human model and the robot are controlled to follow human motion by the processed data synchronously. Furthermore, during the actual operation, the operator can get feedback through the real-world camera and virtual environment. Real-world cameras provide real-world environment feedback, and virtual environments provide virtual world environment feedback from any perspective. The results of this study show that this interaction system can significantly increase the success rate to 80% and reduce the operation time to about 20 seconds during specified tasks. The synchronized human-robot-virtuality interaction system improves both the efficiency and accuracy of teleoperation tasks.