HFR-Video-Based Fingertip Velocimeter for Multifinger Tapping Detection

In this study, we propose a novel concept of a software-based fingertip velocimeter using high-frame-rate (HFR) video processing that can simultaneously estimate when and where an operator taps with his/her finger by detecting the high-frequency component that develops when the fingertip actively contacts something. Our software-based fingertip velocimeter can precisely estimate the velocities of multiple fingers through HFR video processing in real time. Digital image correlation (DIC) operating at every frame for sub-pixel-precision velocity estimation is hybridized with convolution neural network (CNN)-based object detection operating at intervals of dozens of frames to robustly update the fingertip ROI regions during the frame-by-frame DIC operation. We developed a real-time multifinger tapping detection system that can execute DIC operation on
${720}\times {540}$
resolution images at 500 frames/s with CNN-based fingertip detection at 30 frames/s. By presenting several experimental results for finger tapping detection, including virtual keyboard interaction with a ten-finger keyboard input, the effectiveness of our fingertip velocimeter as a finger tapping interface was demonstrated, which can simultaneously estimate the tapping positions and moments of multiple fingers when finger tapping is performed ten times or more in a second.