Augmented Reality-Driven Machine Learning Techniques for Gripper-Integrated Robotics

Learning from Demonstration (LfD) is a widely used method for teaching control policies to robots. However, generating the sensorimotor trajectories used in LfD involves challenges such as the difficulty of robot control and safety concerns due to physical contact. This paper presents an augmented reality (AR) system that eliminates the need for manual intervention during kinesthetic teaching. Unlike previous AR-based LfD systems, our method uses both the user’s hand-drawn spatial trajectory and the real-time open/closed state of the gripper. By synchronizing trajectory data with gripper actions, the system improves the success of tasks like object pick-and-place. Additionally, a deep learning module integrated into the system demonstrates the effectiveness of a model trained on AR-generated data in a ball-in-basket task. The performance of the deep learning method is quantitatively evaluated by comparing it to a standard statistical approach.