Markerless Motion Tracking With Noisy Video and IMU Data

Objective:
Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home.
Methods:
We propose deep learning models to estimate human movement with noisy data from videos (VideoNet), inertial sensors (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration. The video and inertial sensing data used to train the models were generated synthetically from a marker-based motion capture dataset of a broad range of activities and augmented to account for sensor-misplacement and camera-occlusion errors. The models were tested using real data that included walking, jogging, squatting, sit-to-stand, and other activities.
Results:
On calibrated data, IMUNet was as accurate as state-of-the-art models, while VideoNet and FusionNet reduced mean
$\pm$
std root-mean-squared errors by 7.6
$\pm$
5.4
$^{\circ }$
and 5.9
$\pm$
3.3
$^{\circ }$
, respectively. Importantly, all the newly proposed models were less sensitive to noise than existing approaches, reducing errors by up to 14.0
$\pm$
5.3
$^{\circ }$
for sensor-misplacement errors of up to 30.0
$\pm$
13.7
$^{\circ }$
and by up to 7.4
$\pm$
5.5
$^{\circ }$
for joint-center-estimation errors of up to 101.1
$\pm$
11.2 mm, across joints.
Conclusion:
These tools offer clinicians and patients the opportunity to estimate movement with research-grade accuracy, without the need for time-consuming calibration steps or the high costs associated with commercial products such as Theia3D or Xsens, helping democratize the diagnosis, prognosis, and treatment of neuromusculoskeletal conditions.