Real-time Arm Motion Tracking and Hand Gesture Recognition Based on a Single Inertial Measurement Unit

With the development of virtual reality (VR) and augmented reality (AR) devices, handheld controllers and camera-based hand tracking are the most common methods for interacting with the virtual world. This paper proposes arm motion tracking and hand gesture recognition methods based on a single IMU (Inertial Measurement Unit) sensor embedded in a wristband. This approach is more convenient than handheld controllers and more energy-efficient than camera-based tracking. We utilize two deep learning models: one for arm motion tracking and the other for hand gesture recognition. When training the arm motion tracking model, we use two IMU sensors-one placed on the wrist to provide forearm pose data and the other placed above the elbow to provide upper arm pose data. We use the elbow’s IMU data as the reference for training the model and calculate the real-time upper arm pose vector based on the wrist’s pose data. We train the hand gesture recognition model using only the wrist-mounted IMU sensor. This model captures different vibrations in the IMU during various gestures (such as grab, release, snap, and tap) for training. We achieve real-time arm motion tracking and hand gesture recognition using only a single IMU sensor. The angle error for arm motion tracking is only 15 degrees, and the hand gesture recognition accuracy reaches 95%.