generating model and motion extracting from video

[Simon Ammann]

Hello Community

I'm completely new to OpenSim and don't have a lot of understanding of the body. So I apologize in advance for questions that should be obious or clear.

We're evaluating to use OpenSim for a project at my university [ZHaW, Zuerich, Switzerland]. In the end, the goal is to generate data for AI training. The first step is trying to generate motions and add them to a model in OpenSim.

We have videos [front view] of people drinking. What interests us is the movement of the arm and the upper body. We want to determine, whether the person drinking makes normal/healthy movements, or if the movement is somehow compensated due to a stroke or injury.

Now the question arose, if it is somehow possible to generate motions [.mot Files] out of these videos? Can these motions only be generated by motion capture systems, or is there any other way?

Whats more, we would need to fit these motions to an existing model [I'm experimentin with this the Kinematic Arm Model with Articulated Hand: https://simtk.org/frs/?group_id=2230]. Is that even possible and what do we need to pay attention to, when trying to fit a motion to a model?

If this isn' possible, I found a guide on how to create a model in the documentation, but that is very general. Is there a guide on how to create an upper body model with a complete arm and neck?

I'm glad about any help or tipps I can get, so thank you in advance!

[Johannes Amend]

Hello Simon,
I am relatively new to OpenSim too, but I can help you with your first question. For a project at university I use python and the OpenCV libary to estimate poses from videos and extracted the coordinates of joint centers. This worked quite well. You could generate the coordinates like this and run them through a OpenSim model for the motion file.

I hope this has helped you a little.
Best regards
Johannes

[Simon Ammann]

Hi Johannes

That is good news, thank you so much for your response.

I'll defenately look into this. Have you actually managed to import these coordinates into OpenSim? That would of course be an absolute kickstart. If not, I'm happy to share if I make any progress!


Best, Simon