Low-Effort Handheld Device User Authentication Using Musical Sounds

This work proposes a low-effort user authentication system for handheld devices based on active acoustic sensing. Rather than using dedicated acoustic signals, we find common media sounds like music can serve as a sensing signal to verify the phone user’s hand. Specifically, when a notification comes, the smartphone can unobtrusively verify who is holding the device and then decide whether to hide or display the sensitive notification content. Since sound and vibration co-exist, we capture two novel responses via the device’s microphone and accelerometer to describe how the individual’s contacting palm interferes with the two-domain signals, which are then described as time-frequency images and fed into a convolutional neural network-based algorithm for user authentication. Moreover, we develop a cross-domain method to validate the hard-to-forge physical relationships among the smartphone’s microphone, speaker, and accelerometer, which are embedded on the same motherboard. This prevents external sounds from cheating the system. Additionally, we consider vibration alerts as a special type of musical sound and extend our method to work with the smartphone’s silent mode. Extensive experiments with ten musical sounds and five phone models show that our method verifies users with 94.5% accuracy and effectively prevents acoustic replay attacks and physical hand forgeries.