Mixed Reality Framework for Eye-in-Hand Collaborative Robot-Human Interaction

In recent years, the manufacturing sector has undergone a significant transition from a paradigm of robotic autonomy to human-robot collaboration. This approach allows operators from different domains to rapidly adapt to a new way of working and reduces redundant learning costs, while also enabling intuitive robot control. To further enhance this paradigm, integrating Mixed Reality (MR) technology has emerged as a promising approach to creating and working in a virtual workspace while having a safe environment for training. This work develops a framework, based on digital twin representation, to integrate the MR with real-life equipment and humans. This allows for the control of a collaborative robot in an environment with objects or humans perceived by the eye-in-hand visual sensor. The efficiency of the proposed framework is emphasized by a comparison of Unity simulations versus the real-time behavior of the collaborative robot.