Stealthy and Practical Multi-modal Attacks on Mixed Reality Tracking

Mixed Reality (MR) is rapidly becoming an essential technology for critical applications such as physical therapy and surgery, in addition to its early use for leisure activities. This transition necessitates a focused look at the security aspects of MR devices and applications. Prior work on MR security focuses on generic aspects such as secure authentication and vulnerability analysis. However, MR devices are multi-modal and spatiotemporal, which exposes them to attacks on sensor modalities across spatiotemporal axes. Prior work has demonstrated attacks on individual sensing streams, but modern, state-of-the-art sensor fusion-based tracking algorithms can easily mitigate such attacks.In this paper, we introduce a practical attack surface; it simultaneously launches attacks on multiple sensing streams across spatiotemporal axes to yield effective, stealthy, and precise outcomes. To the best of our knowledge, our work is the first to propose, design, and evaluate simultaneous multi-modal spatiotemporal attacks. In doing so, we solve key challenges in deciding what attack mechanisms to use, when to launch attacks, and how to configure attacks. Using the tracking and navigation use case of a user wearing an MR headset in real-world settings, we demonstrate the effectiveness of our attacks over the user’s trajectory in the presence of state-of-the-art sensor fusion-based tracking algorithms and system checks.