MirrorTrack — a real-time multiple camera approach for multi-touch interactions on glossy display surfaces

This paper presents a real-time multiple camera approach for multi-touch interaction system that takes advantage of specular display surface (such as conventional LCD displays) and the mirror-effect in a low-azimuth camera angle to detect and track fingers their reflections simultaneously. Building on our prior work, 1. We use multi-resolution processing to greatly improve runtime performance of the system; 2. We employ different edge detection and pattern recognition algorithms for different processing resolution to help detect fingers more accurately and efficiently; 3. We track both the location of a fingertip and its pointing direction so it can be identified more effectively; 4. We use a full stereo algorithm to compute finger locations in the 3D space more accurately. Our system has many advantages. 1. It works with any glossy flat panel display; 2. It avoids clumsy set-up time of a top-down camera with the concomitant screen glare problems; 3. It supports both touch and hover operation; 4. It can work with large vertical display without the usual occlusion problems. We describe our approach and implementation in details.