Typing on Any Surface: Real-Time Keystroke Detection in Augmented Reality

The ineffectiveness of text entry interfaces remains a significant barrier to social engagement in augmented reality (AR). Popular options, such as mid-air keyboard interface, wireless keyboards or voice input face challenges such as ergonomic issues, limited accuracy, or social discomfort in public use. This paper introduces a deep-learning method allowing AR applications to predict keystrokes from the user perspective video stream captured by any AR headset. This enables users to type on flat surfaces without a physical or virtual keyboard. Our two-stage model combines an off-the-shelf hand landmark extractor with an innovative adaptive Convolutional Recurrent Neural Network (C-RNN). It was trained on a newly built dataset, enabling prediction of 27 keys (alphabet and space) at approximately 32 FPS. With practice, users can reach a 91.0% accuracy at 40 words per minute, comparable to typing on a physical keyboard. The encouraging results demonstrate our method’s feasibility and potential for integration in diverse applications. We also explore limitations and future research directions for production system implementation.