Flying a toy plane

The availability of powerful computing and inexpensive cameras is engendering a new era of real-time consumer-grade computer vision. We have devised a vision system for cheap, unencumbered user input with six degrees of freedom and have applied it to virtual flight on a typical consumer platform. Sitting before a monitor-mounted camera, a user can intuitively pilot a flight simulator by relative position (X, Y and Z) and orientation (roll, pitch, and yaw) of a specially-marked toy plane held in the hand. This paper describes the design, acquisition, tracking, and pose estimation of our object model. We have achieved a robust scheme capable of operating in a demanding environment with unpredictable lights, backgrounds, and camera properties. The success of our approach relies heavily on tracking. We allow multiple hypotheses and benefit from the intensity invariance and fast, semiglobal trackability of the cliff feature, which we introduce as a useful feature abstraction. While this work demonstrates an integrated approach for solving a specific problem, the ideas can be applied elsewhere.