Enhancing Inertial Measurement Unit Based Posture and Motion Estimation through Multi-Modal Training Using Video Posture Detection Models

This paper presents a deep learning approach for estimating human joint and segment angles directly from wearable inertial sensor data using a sequence-to-regression framework. The proposed architecture employs a stacked Long Short-Term Memory (LSTM) network to model temporal dependencies in multivariate Inertial Measurement Unit (IMU) signals, followed by a sequence of fully connected layers to regress a full-body pose representation. The model is trained end-to-end using IMU signals as input and joint and segment angle targets extracted from video via 2D pose estimation using the Sports2D framework. During inference, the model predicts full-body pose using only IMU data, without relying on video or pose estimation tools. Experimental results show that the model accurately predicts full-body joint and segment angles from IMU data, with strong generalization to unseen motion sequences. Evaluation confirms reliable performance across diverse movements, particularly in lower-limb segments, while identifying higher variability in upper-body joints. These findings support the efficacy of deep recurrent networks for direct IMU-to-pose estimation and provide a foundation for extensions incorporating attention mechanisms, sensor fusion strategies, and biomechanically informed learning objectives.