Developing Ubiquitous Multi-touch Sensing and Displaying Systems with Vision-Based Finger Detection and Event Identification Techniques

This study presents efficient vision-based finger detection, tracking, and event identification techniques, as well as a low-cost hardware framework for multi-touch sensing and display applications. A fast bright-blob segmentation process based on automatic multilevel histogram thresholding is performed to extract pixels of touch blobs from the captured image sequences obtained from the scattered infrared lights by the video camera. Given the touch blobs extracted from each of the captured frames, a blob tracking and event recognition process is then conducted to analyze the spatial and temporal information of these touch blobs from consecutive frames and determine the possible touch events issued by users. This process also refines the detection results and corrects for errors and occlusions caused by noise and errors during the blob extraction processes. Our proposed blob tracking and touch event recognition process includes two phases. First, the phase of blob tracking associates the motion correspondence of blobs in succeeding frames by analyzing their spatial and temporal features. Then the phase of touch event recognition process can identify meaningful touch events activated by users from the motion information of touch blobs. Experimental results demonstrate that the proposed vision-based finger detection, tracking, and event identification system is feasible and effective for multi-touch sensing applications in various operational environments and conditions.