Mapping 2D input to 3D immersive spatial augmented reality

This poster presents Viewpoint Cursor, a technique for mapping 2D user input from devices such as mobile phones, trackballs, or computer mice, to 3D multi-projector spatial augmented reality systems. While the ubiquity of input devices such as these make them obvious choices for spatial augmented reality, their 2D nature makes them difficult to use. Existing VR techniques rely on a display in front of the user's eyes on which to place virtual information. Immersive spatial augmented reality systems allow users to experience and interact with projected virtual information from any angle, using arbitrary placement of projectors. Viewpoint Cursor addresses these issues by mapping 2D input to a plane in front of the user's view. Ray casting is then used to find the 3D location for the cursor in the scene, which is then projected using the projection system. The user's position is tracked, with the input remapped accordingly, resulting in 2D input that matches what the user expects, regardless of their location.

1 Introduction

This poster presents Viewpoint Cursor, a new interaction technique for Spatial Augmented Reality (SAR) systems. Viewpoint Cursor maps 2D user input to a 3D SAR environment, allowing users to intuitively select items and manipulate virtual information using common 2D input devices. SAR presents unique challenges to researchers, including the lack of an image plane in the user's field of view and the need for all virtual information to appear on surfaces in the environment. The ubiquity of devices such as the mouse, hand-held trackballs, and mobile phones make them compelling choices for SAR. However, standard cursor mapping is unsuitable, as it was designed for monitors placed side by side directly in front of users. SAR often uses projectors facing one another in order to project onto all sides of an object. This causes significant difficulty when selecting objects, as the user must move the cursor to the correct projector. In addition, SAR allows the user to view and interact with the scene from any angle, making 2D input difficult for the user to understand. Viewpoint Cursor solves these problems by remapping 2D input based on the user's view of the scene. Moving left with the input device will always move the cursor left, from their point of view. Cursor placement is accomplished using a ray-casting algorithm independent of projector locations. This removes the need to move a cursor into the correct projector's frustum. The User controls the cursor using a touch-pad app on a mobile phone, as shown in Figure 1. Figure 1

A user interacts with a SAR system using viewpoint cursor and a mobile phone.