Conda OpenSim slow on Linux

[Pagnon David]

Hi,

I installed the opensim conda package, and I noticed that inverse kinematics is much slower on my Ubuntu server than on my laptop. This is annoying, since I need to process a massive amount of data that I cannot download on my computer.

Do you have any idea why, and how to speed it up?

Here are my specs:
LAPTOP:

Windows 11
CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz 2.59 GHz (*12)
GPUs: NVIDIA Quadro P1000
RAM: 32 Go
python -V = 3.9.19
opensim.__version__ = 4.5

SERVER:

Ubuntu 22.04.4 LTS
CPUs: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz (*16)
GPUs: GeForce GTX 1080 Ti, GeForce GTX 1070,GeForce RTX 2080
RAM: 251Gi Mem, 251Gi Swap
python -V = 3.11.9
opensim.__version__ = 4.5.1

My laptop is about 6-7 times faster than the server on a simple IK task.
I first tried with Python 3.8 and OpenSim 4.4.1, with the same results.

Thank you in advance!

[Ayman Habib]

Hi David,

Thanks for reporting. Let me start by clarifying that, as it stands, OpenSim (Outside of Moco) is single threaded, as such the fact that the server has multiple cores or threads would have no bearing on performance.

Running the same version of OpenSim/conda package would factor out differences due to code changes/python/numpy version differences if any and would help us identify if there're actual significant differences in code performance vs. differences in settings/environment.

That said, if you are processing a large dataset, you should be able to parallelize the inverse kinematics for different trials/subjects/elements among multiple cores entirely on your side by creating and managing threads in python.

Hope this helps and please let us know what you find out.
-Ayman

[Pagnon David]

Hi Ayman,

Thank you for answering!
First of all, you're right, I'll try to parallelize IK on my multiple threads, it will run significantly faster.

Considering that python 3.8 with OpenSim 4.4.1 and python 3.11.9 with OpenSim 4.5.1 ran at the exact same (slow) speed on my linux system, I would not expect it to be very different for python 3.9.19 with Opensim 4.5.
I suppose my Windows CPU is very slightly faster than the Linux one, but I would not expect it to make a significant difference. Although I'm not an expert at all in that field, and there may be differences that I don't understand. I'm not sure that RAM or GPUs have anything to do with the computation speed. Aside from these parameters, the main difference is the OS.

[Pagnon David]

I'm just remembering something that could be of significant importance: I first could not run OpenSim IK, as it made Python crash with this error:

Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so

I solver the issue by doing:

Code: Select all

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libmkl_def.so:/usr/lib/x86_64-linux-gnu/libmkl_avx2.so:/usr/lib/x86_64-linux-gnu/libmkl_core.so:/usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so:/usr/lib/x86_64-linux-gnu/libmkl_intel_thread.so:/usr/lib/x86_64-linux-gnu/libiomp5.so

Then I stopped thinking about it. Do you think it could have anything to do with my issue? (and also, how to solve it?)

Although the user uses Matlab, it might be related to:
- viewtopicPhpbb.php?f=91&t=7794&p=20730&start=0&view=
- https://github.com/opensim-org/opensim-core/issues/3423

[Pagnon David]

Okay I think I got it:
- On Windows, the opensim conda package is shipped with mkl, mkl_fft, mkl_random
- On Linux, mkl_fft and mkl_random are removed, it installs nomkl, libopenblas, openblas instead (numpy is also downgraded)

However, it does not seem like OpenBLAS accelerates linear algebra as much (computation speed is about 7 times slower). More information there: https://numpy.org/install/#numpy-packag ... -libraries

Out of curiosity, what prevents you from using MKL?