OpenSense OpenSim IMU placer | inverse kinematics tool question

[Fangshi Zhu]

Dear all,

The newly integrated OpenSense (IMU placer and IMU inverse kinematics) tool is great. I tried it with our IMU data and was able to get some decent ik results except for one issue: when the subject turns around for 180 degrees and walks along the opposite direction of the OpenSim +X (starting direction/first frame direction), the IMU ik-driven model doesn't seem to compensate the turning and look like it is walking backward.

In the OpenSim wiki example, the (Xsens) IMU ik result seems correct even when the subject walks around and turns. However, I had a hard time getting it right with our data when the subject turns 180 degrees. Could someone provide some help this?

----------------------------------------
---- Below are the detailed description of the problem, and a test dataset (including model.osim, IMU quaternion.sto, IMU placer XML, IMU_ik.mot) -------------------------------------------


Brief question description:
IMU inverse kinematics results look wrong when the subject turns/walks to the opposite direction of the starting direction (OpenSim +X).

Detailed description:
Method:
We used 7 Alubi IMU sensors (local axis direction shown below) and recorded the sensor quaternions from each sensor. The data format, and organizations are not the same as the Xsens sensors/system, so I wrote custom scripts to convert our data into the 'xxx_quaternion.sto' format (single, time synced, quaternions storage (.sto) file) that is compatible with the OpenSim|OpenSense software.

The sensor (blue cubes are Alubi IMU sensors) attachment and its local coordinate system are shown below:
https://www.dropbox.com/s/1dr6g5wb4ctvc ... 1.png?dl=0

My processing steps are:

• I first used the 'IMU placer tool' to place IMU on the subject model. (Align IMU: pelvis_imu, heading correction: +Z; Rotation: (90, 0, 0), as shown above, to match the OpenSim coordinate axis direction).

• Then I used the 'IMU Inverse Kinematics' (Rotation: (90, 0, 0)) tool to run inverse kinematics computation.

The motion task was: subject standing straight in neural posture in the beginning (first frame used in IMU placer calibration), then walking back and force between point A and point B for 2 times:
https://www.dropbox.com/s/uwwklx0evmp0f ... 2.png?dl=0

As in starting position, the subject|pelvis_imu sensor +Z direction is facing the OpenSim +X direction, when using the IMU placer tool, I did the sensor alignment & heading correction as pelvis_imu, +Z, rotation(90, 0, 0) .

The motion task in the entire trial task was:

• Walk along OpenSim +X direction

• Walk along OpenSim -X direction

• Walk along OpenSim +X direction

• Walk along OpenSim -X direction

• Turn around to face OpenSim +x direction again.

The inverse kinematics result computed using the OpenSim IMU Inverse Kinematics tool was problematic:

• When the walking direction is the same as the starting position direction, the result seems correct (from OpenSim model 3D animation)

• When the walking direction is the opposite of the starting position direction, the result seems incorrect (as if the subject walks backward).

In the example (OpenSense - Kinematics with IMU Data) on the OpenSim Wiki page: https://simtk-confluence.stanford.edu/d ... h+IMU+Data
The IK result looks correct across the entire trial. Even after the subject turns around, all the leg joints still flex/extend in the right direction:
https://www.dropbox.com/s/c8pw9kmpb93xv ... 3.png?dl=0

However, in our case, after the subject turns around 180 deg, the results look like the subject was walking backward. I did some tests and found that when placing the IMU (using the IMU Placer Tool), by simply flipping the Heading correction from pelvis_imu, +Z direction to pelvis_imu, -Z direction, the results (right/wrong) direction also flipped. Now:

• when the subject walks in the opposite of starting direction, the result is correct

• when walking along the starting direction, the result looks wrong (walk backward).

https://www.dropbox.com/s/bihtal6l3diwp ... 4.png?dl=0
https://www.dropbox.com/s/30v19a3qjvvhg ... 5.png?dl=0

For your reference, here is a video clip of the results I got:
https://www.dropbox.com/s/bt74u6rcwijri ... l.mp4?dl=0

Could you please provide some help on this matter? How to make sure the subject turning does not affect the IMU ik result?

FYI, I have attached our model, IMU placing and ik setup file (.xml), a test dataset (.sto), and ik results (.mot) for you to test out:
https://www.dropbox.com/sh/fpsk7gp5jutg ... JS-fa?dl=0

Here is the folder organization structure:
https://www.dropbox.com/s/tnqpemhs5zt20 ... 6.png?dl=0

Any suggestion or comment is appreciated.

---
Fangshi Zhu, Ph.D.
Postdoctoral Research Fellow
NeuroRecovery Research Center
Department of Physical Medicine and Rehabilitation
University of Texas Health Science Center at Houston
1333 Moursund St
Houston, TX 77030
Email: Fangshi.Zhu@uth.tmc.edu | fungshi.zhu@gmail.com

[Ayman Habib]

Hi Fangshi,

Thanks for the detailed description and thorough investigation. I noticed that you were using OpenSim 4.1. Before digging deeper into it, I'd suggest you try version 4.2 which has some bug fixes to the calibration procedure and offers tools for visualization of sensor data e.g.
https://simtk-confluence.stanford.edu/d ... ensor+Data

Please give that a try and let us know what you find out. General guidelines with the version 4.2+ OpenSense code:
1. The calibration process is not supposed to be a trial and error process, if you know the direction facing forward and the orientation of the base IMU this should be pretty straightforward.
2. If you load the model after IMU Placement in the GUI and see the IMU frames have the correct orientation and that they are on the correct segments that match your data collection then IMU Placement worked correctly and you don't need to touch it again. Typically this is the more error prone step.
3. You can use the preview sensor data option (documentation link above) to see if the raw data is correct or if it's an issue with data collection/conversion/manipulation. Our example uses XSens format/convention but if you have your own custom IMU that computes orientation/quaternions differently then you need to account for that before using OpenSense.
4. If some sensors look bad or suspect for some reason (e.g. malfunction or influenced by magnetic field) then you can exclude them from IK using weights.

Hope this helps, and please let us know what you find out.

Best regards,
-Ayman

[Fangshi Zhu]

Thank you for your fast replying, Dr. Habib.

To clarify, I used OpenSim 4.2 (not version 4.1).

I tested each Alubi IMU and confirmed that all sensors work correctly and accurately (give correct quaternion/sensor orientation as I rotate the sensor around its x/y/z axis).

I tried preview sensor data (7 IMUs orientation), and they seem to match the real movement during the experiment task (small pitching during straight-line walking, and rotate/yaw 180 degrees when turning).

Despite the detailed 'OpenSense - Kinematics with IMU Data example' on the wiki page (https://simtk-confluence.stanford.edu/d ... h+IMU+Data), I am still a bit confused about the heading correction and sensor rotation setting as it is not clearly described.

My understanding is, when placing IMU/calibrating the model, there are two steps needed to align the sensor local coordinate axis (in our case, forward (Z), right (X), up (Y)) to the OpenSense global axis (forward (X), up (Y), left (Z)):

• Specify the base IMU and its heading direction. In our case, it is base: pelvis_imu, heading: +Z; Then the OpenSim will re-label the original IMU local axis, following the right-hand rule, starting with the replacing the old 'Z' (heading) direction with new 'X', so
  IMU old: forward (Z), right (X), up (Y) ---> IMU_heading_corrected: forward (X), right (Y), right (Z)

• Then, the heading corrected IMU sensor coordinate axis (forward (X), right (Y), right (Z)) needs to be rotated to align with the OpenSim axis (forward (X), up (Y), left (Z))

• • IMU_heading_corrected: forward (X), right (Y), right (Z) ------- rotate clockwise around corrected forward (X) axis for 90 degree --->
    IMU_heading_corrected_rotated: forward (X), right (Y), right (Z)

So, in the IMU placer/model calibration process, the setup was:

• Base IMU: pelvis_imu
  Heading: +z
  Sensor rotation (x, y, z): (+90, 0, 0)

Then when performing "IMU inverse kinematics", the setting is:

• Sensor rotation: (+90, 0, 0)

Here is a picture showing our sensor attachment location, OpenSim coordinates, IMU original coordinate axis and the two-step alignment process:
https://www.dropbox.com/s/ke2hfi1au8lpn ... K.png?dl=0

Please let me know if this IMU placer and IK tool setup are correct. If not, how should I set up our IMU (default axis: forward (Z), right (X), up (Y)) correctly?

---------------
Assuming the IMU placing and IK setup were correct, I ran the IMU inverse kinematic calculation on one motion trial, where the tasks are:

• T1: First frame: standing still in neutral posture.

• T2: Then walk straight forward for about 20 ft (along the first frame direction).

• T9: Turn around for 180 degrees.

• T4: Then walk straight back for about 20 ft (opposite of the first frame direction).

• T9: Turn around for 180 degrees.

• T6: Then walk straight forward for about 20 ft (along the first frame direction).

• T9: Turn around for 180 degrees.

• T8: Then walk straight back for about 20 ft (opposite of the first frame direction).

• T9: Turn around for 180 degrees.

I found from the result that, if the IMU placing setup were:

• Base IMU: pelvis_imu
  Heading: +z (forward with respect to the human model in the first frame)
  Sensor rotation (x, y, z): (+90, 0, 0)

The ik results were:

• The task T2 and T4 are walking along the same heading direction, and their results look correct;

• The task T4 and T6 are walking along the opposite of heading direction, and their ik result look wrong.

-------
However, if the IMU placing setup were:

• Base IMU: pelvis_imu
  Heading: -z (backward with respect to the human model in the first frame)
  Sensor rotation (x, y, z): (+90, 0, 0)

The the ik results were:

• The task T2 and T4 are walking along the opposite of heading direction, and their results look wrong;

• The task T4 and T6 are walking along the same heading direction, and their ik results look correct.

Here is a video demo of our IMU attachment location, IMU sensor local axis, OpenSim axis, IMU placer setup and IK results:
https://www.dropbox.com/s/bt74u6rcwijri ... l.mp4?dl=0

I do not understand why when the subject turns, the IK results are wrong. In the 'OpenSense - Kinematics with IMU Data example' on the wiki page (https://simtk-confluence.stanford.edu/d ... h+IMU+Data), the IMU ik seems to handle the subject turning very well. However, in our case, when the subject does not walk along the corrected heading direction, the result is wrong.

I tried everything I can and read the online tutorial many times, and could not get what I did wrong. Could you please take a look at our data? I have attached our model, IMU placing and ik setup file (.xml), a test dataset (.sto), and ik results (.mot) for you to test out:
https://www.dropbox.com/sh/fpsk7gp5jutg ... JS-fa?dl=0

Here is the folder organization structure:
https://www.dropbox.com/s/tnqpemhs5zt20 ... 6.png?dl=0

Thank you very much!