Motion Generation and Analyzing the User’s Arm Muscles via Leap Motion and Its Data-Driven Representations

In this study, we introduce a novel framework for practicing and analyzing arm muscles in motions, such as juggling actions, by estimating the hand motion of the user using a Leap Motion device. The proposed method can map the movement of a ball in a virtual world to the hand motion of the user in real time and visualize the relaxation and contraction of muscles to determine the amount of exercise performed. Our procedure has five sections: 1) The Leap Motion device tracks the hand position of the user. 2) A behavioral pattern in which a user throws a ball is defined as an event. 3) The hand motion is mapped to the ball based on the hand position of the user using the proposed parabola-based particle approach. 4) The quantity of muscle activity is visualized and analyzed in relation to the degree of arm bending. 5) Finally, we propose a method that utilizes the symmetry data-driven approach to extend solvers, enabling the efficient handling of avatar juggling motions in a virtual environment, based on user actions. Moreover, this method enhances the results by allowing diverse control over the virtual ball’s trajectory to match the user’s pose. Consequently, the proposed system enables real-time juggling in a virtual environment, as well as practice and analysis of the arm muscle activity of the user. The outcomes of the analyses are expected to be applied in various industries, including healthcare. In the solver extensions, we do not utilize all the hand position information of the user. Instead, we base the trajectory of one hand on the data of the other hand, synthesizing it through a data-driven approach. This results in a relatively lightweight algorithm that generates the avatar’s juggling motion. Additionally, by leveraging the user’s pose and parabolic motion, we can arbitrarily synthesize the trajectory of the virtual ball, facilitating the easy creation of a variety of scenes.