CasePad: Privacy-preserving Finger Activity Sensing via Passive Acoustic Signals Enhanced by Mini-Structures in Smartphone Cases

Smartphones have emerged as indispensable devices, seamlessly integrating into our daily lives. However, traditional smartphone interfaces, primarily relying on touchscreens, raise privacy concerns and are susceptible to privacy leakages. We thus propose CasePad, an innovative system that leverages low-cost smartphone cases to achieve fine-grained finger activity sensing while preserving users’ privacy. Toward this end, we devise a passive system to exploit acoustic signals generated from finger interactions on the back of the smartphone case. Our novel approach leverages acoustic mini-structures embedded within the smartphone case to regulate the acoustic signals from finger interactions and enhance their diversity. We further develop a multi-task learning framework including a multi-scale shared encoder and task-specific decoders to extract comprehensive acoustic features of finger activities. To achieve precise predictions, we utilize the Multilayer Perceptron (MLP) as an encoder and design a series of loss functions in decoding tailored to the specific characteristics of finger activities. During the offline training, CasePad utilizes raw passive finger activity sound as input and leverages the camera for supervision. With the use of Siamese network to extract feature files that only contain finger activity-specific information, the user does not need to collect data to train their own model. Extensive experimental evaluations with different smartphone models validate CasePad’s high performance, achieving 98.76% classification accuracy in detecting finger activity direction. Additionally, CasePad demonstrates remarkable precision in deriving detailed finger activity characteristics that closely match the ground truth measurements across various finger activities, including position tracking with a mean squared error (MSE) of 10.28 mm, distance estimation with an MSE of 9.32 mm, and speed derivation with a mean absolute error (MAE) of 7.29 mm/s, respectively.