A Registration Method for a Mixed Reality System and its Application to Robot Path Generation

This paper presents a robot path generation system based on Mixed Reality (MR) technology. We propose an on-site registration method based on the posture of the manipulator to enhance the path execution. A user wears MR glasses to obtain environmental information, and the point cloud segmentation algorithm determines the manipulator pose. On the other hand, the STL model and real-time controller data are used to generate the current point cloud of the manipulator posture. The Simple-ICP algorithm is applied to align the MR-based and robot-based coordinate systems to obtain the registration result. Based on the result, the manipulator can execute the path defined in the virtual environment by hand gestures. The proposed method has been implemented and verified using a TM5 collaborative robot and HoloLens II. Experimental results show that the registration achieves an RMSE within 3.37 mm, an average image projection error within 4.82 mm, and an average path execution error within 7.73 mm.