Lift: Using projected coded light for finger tracking and device augmentation

We present Lift, a visible light-enabled finger tracking and object localization technique that allows users to perform freestyle multi-touch gestures on any object's surface in an everyday environment. By projecting encoded visible patterns onto an object's surface (e.g. paper, display, or table), and localizing the user's fingers with light sensors, Lift offers users a richer interactive space than the device's existing interfaces. Additionally, everyday objects can be augmented by attaching sensor units onto their surface to accept multi-touch gesture input. We also present two applications as proof of concept. Finally, results from our experiments indicate that Lift can localize ten fingers simultaneously with an average accuracy of 1.7 millimeter and an average refresh rate of 84 Hz with 31 milliseconds delay on WiFi and 23 milliseconds delay on serial communication, making gesture recognition on non-instrumented objects possible.