Remote Monitoring and Control of Mobile Robots in Real-Time Using Multimodal Digital Twins

Although digital twins have been playing a pivotal role in the management of the lifecycle of physical robotic systems of systems, they have hardly been employed to actively guide mobile robots in real-time. In fact, the guidance of such systems requires functionalities, including the perception of environmental constraints and avoidance of collisions. A spatial information stream beyond the internal robot state measured using proprioceptive sensors is therefore needed. Exteroceptive sensors help meet this demand. Nevertheless, such sensors have received little attention thus far in the development of digital twins. On the other hand, various mobility objectives, such as reverse motions, might require the awareness of the historical internal state of the distant robot. For instance, the current energy budget is likely to constrain the reachability of the initial robot location, even when spatially and kinematically feasible. We therefore develop a web-based framework to monitor and actively steer mobile robots while leveraging on multimodal digital twins. We collect data about the internal state and camera-captured neighborhood of the robot in real-time. The robot operator is thereby provided with an enriched digital model of the internal dynamics and environmental perception of the robot. This elevates situational awareness and facilitates decision support. We then develop a versatile graphical interface that helps holistically monitor and actively steer the mobile robot. Since the multi-modal and bidirectional approach is intuitive and device-agnostic, even novices can remotely guide the robot on mobile phones from everywhere. We show the usefulness and effectiveness of our approach in use-case scenarios in practice.