Programming by Demonstration using Mixed Reality and Simulated Kinematics

abstract-Amidst the rapidly evolving landscape of human-robot interaction, Programming by Demonstration (PbD) emerges as a promising avenue for intuitively instructing robotic systems. This paper presents a groundbreaking fusion of Mixed Reality (MR) technology and simulated kinematics, aimed at transforming the robotics programming paradigm. Through the integration of MR interfaces and simulated kinematics, this project seeks to democratize robotics programming, ensuring accessibility across diverse user skill levels, and enhance safety in robotic environments. Leveraging the immersive capabilities of the Microsoft HoloLens 2 headset, users can engage with virtual representations of robots in real-world environments. Simulated kinematics complements this interaction by offering a digital twin of the robot, facilitating visualization and validation of instructions prior to execution. The paper delves into the hardware configuration, software integration, user interface design, encountered challenges, and potential applications of this innovative approach. By harnessing the relationship between MR technology and simulated kinematics, this project addresses longstanding barriers in traditional robotics programming, ushering in a new era of intuitive human-robot interaction across various domains.