Hands-free Mobile Device Control Through Head Pose Estimation

Despite touch input being the predominant way to interact with mobile devices, there are many situations where alternative forms of input are required. Touch input is not only problematic for users with motor impairments, but also inconvenient when the hands are oily from cooking or needed to hold something. Existing alternatives like voice input or hand gestures only mitigate some of the problems. They raise privacy concerns when others can overhear the voice commands and are compromised by inaccurate sensor readings in highly dynamic mobile contexts. In silent environments like libraries, voice input is not an option either. In this paper, we therefore design and implement RUDoLF, a robust control interface for mobile devices based on head poses. Users can turn their head in four directions to control applications. To classify head poses, we extract facial landmarks from video frames captured with the front facing camera. Our evaluation with three mobile applications validates RUDoLF as a robust and easy-to-use control interface that outperforms previous approaches in speed, accuracy, and usability.