Inertial sensing and computer vision are promising alternatives to traditional optical motion tracking, but until now these data sources have been explored either in isolation or fused via unconstrained optimization, which may not take full advantage of their complementary strengths. By adding physiological plausibility and dynamical robustness to a proposed solution, biomechanical modeling may enable better fusion than unconstrained optimization. To test this hypothesis, we fused RGB video and inertial sensing data via dynamic optimization with a nine degree-of-freedom model and investigated when this approach outperforms video-only, inertial-sensing-only, and unconstrained-fusion methods.

This project links to a repository containing code for running 4 classes of simulations in MATLAB with a nine DOF biomechanical model for estimating full body kinematics (and dynamics and contact forces if using direct collocation):

(1) IMU and vision data fusion (tracking) via direct collocation

(2) IMU only data tracking (and denoising) via direct collocation

(3) Unconstrained IMU and vision data fusion via inverse kinematics

(4) Unconstrained kinematics calculations using computer vision keypoints only

The code is set up to run on sample experimental data from the article below as well as synthetic created from sample walking mocap data contained in open source software OpenSim's installation documents. Users can validate findings from the referenced article with the sample experimental data as well as explore methodological trade-offs using synthetically generated data with sample mocap data. However, it should be noted that the findings of the study cannot be validated using the synthetically generated data from mocap data since they incorporate simplified assumptions and noise models that are not as representative of the true noise backgrounds of experimental IMU and computer vision data used to generated the findings of the referenced article. The synthetic approach is provided for convenience and learning. This synthetic approach was NOT utilized to generate the findings of the referenced article; it also overestimates the accuracy of each approach. However, it does still illustrate the trade-offs accurately.