Enhancing Human-in-the-Loop Adaptive Systems through Digital Twins and VR Interfaces

Self-adaptation approaches usually rely on closed-loop controllers that avoid human intervention from adaptation. While such fully automated approaches have proven successful in many application domains, there are situations where human involvement in the adaptation process is beneficial or even necessary. For such “human-in-the-loop” adaptive systems, two major challenges, namely transparency and controllability, have to be addressed to include the human in the self-adaptation loop. Transparency means that relevant context information about the adaptive systems and its context is represented based on a digital twin enabling the human an immersive and realistic view. Concerning controllability, the decision-making and adaptation operations should be managed in a natural and interactive way. As existing human-in-the-loop adaptation approaches do not fully cover these aspects, we investigate alternative human-in-the-loop strategies by using a combination of digital twins and virtual reality (VR) interfaces. Based on the concept of the digital twin, we represent a self-adaptive system and its respective context in a virtual environment. With the help of a VR interface, we support an immersive and realistic human involvement in the self-adaptation loop by mirroring the physical entities of the real world to the VR interface. For integrating the human in the decision-making and adaptation process, we have implemented and analyzed two different human-in-the-loop strategies in VR: a procedural control where the human can control the decision making-process and adaptations through VR interactions (human-controlled) and a declarative control where the human specifies the goal state and the configuration is delegated to an AI planner (mixed-initiative). We illustrate and evaluate our approach based on an autonomic robot system that is accessible and controlled through a VR interface.