4 Degrees of Freedom: Spatial Tracking and Sensing

Chapter 3 was about the space that computer graphics constructs; this one is about finding the user of an AR or VR system in space. The two questions here are: Where is the user in the world (spatial tracking), and what is the world like around the user (spatial sensing)? The first is the key to making an AR or VR experience responsive. The user can effectively examine and move through the space of the experience only if the computer can keep track of where she is and where she is looking. The second question is the key to a new set of AR (and potentially VR) applications. Improved cameras and software for computer vision are making it possible for mobile devices to "understand" the world that the cameras see. Applications begin to see into the world, marking surfaces such as floors and walls in three dimensions and eventually being able to distinguish furniture, humans, and all the other objects the room contains.