Codeless, Inclusive, and End-to-End Robotized Manipulations by Leveraging Extended Reality and Digital Twin Technologies

Although the importance of robots for efficient production in smart factories and assistance in home settings is well recognized, their usage in industry and society still faces challenges. A cause lies in the difficulty that comes with programming robots by writing several lines of code to complete basic tasks. Unfortunately, workers in particular as well as citizens in general rarely have the necessary technical background and experience to achieve this complex goal. Recruiting professional robot programmers creates a dependency and increases costs associated with production and assistance. This issue gets more complicated and urgent when robots are deployed to accommodate small batch sizes and heterogeneous preferences from Industry 4.0 and Industry 5.0 applications. In fact, the duration allocated to the changeover in robotized applications might be brief to meet concurrent economic objectives. Furthermore, generic motion primitives provided by robot vendors might not match with irregular and domain-specific Cartesian trajectories in industry, society, and in-between. This work embraces these issues by integrating extended reality and digital twins to propose an intuitive, inclusive, fast, and effective framework to program (i.e., plan and execute) robot motions. Novices can thereby move a manipulator to reach freely specified Cartesian frames and complete grasping maneuvers. This is done without writing any line of code. How our approach empowers citizens is shown by completing manipulations to grasp and relocate workpieces in practice.