Progress Observation in Augmented Reality Assembly Tutorials Using Dynamic Hand Gesture Recognition

We propose a proof-of-concept augmented reality assembly tutorial application that uses a video-see-through headset to guide the user through assembly instruction steps. It is solely controlled by observing the user's physical interactions with the workpiece. The tutorial progresses automatically, making use of hand gesture classification to estimate the progression to the next instruction. For dynamic hand gesture classification, we integrate a neural network module to classify the user's hand movement in real time. We evaluate the learned model used in our application to provide insights into the performance of implicit gestural interactions.