Get Ipopt to use mumps vs pardiso

[Pavlos Silvestros]

Hello everyone,

With some help I have managed to get OpenSim Moco running on some clusters I have access to. In my tests on the cluster (built from source) I get my optimal control problems to run however I have noticed that Ipopt is using pardiso. This is different to my local machines (not built from source) that use mumps and also the function error when using pardiso is at least 100x larger for the same sims when using pardiso vs mumps.

I have tested this on both Intel and AMD processors, and tried loading specific mumps-metis libraries after installation as a last ditch attempt but still encountering it. Is there a way to specify mumps during runtime or is there a flag that I missed during compiling/building?

As I am writing this I realised that my local machines are running OpenSim 4.3 whereas I installed 4.4.... Before I do everything again, would you expect that to have an effect due to an update?

I see they are a couple more threads related to mumps and HPCs but couldn't quite understand to to apply the answers to my problem.

Thanks as always!

Cheers,
Pavlos

--Some info on the cluster system

Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.80.1.el7.x86_64
Architecture: x86-64

[Nicholas Bianco]

Hi Pavlos,

Here's the CMake for building Ipopt on Unix machines: https://github.com/opensim-org/opensim- ... s.txt#L318.

The patch file for getting MUMPS is located in the same directory in the source.

MUMPS requires gfortran. Have you double checked that you're installing it (brew install gcc) on your cluster machine?

Best,
Nick

[Nicholas Bianco]

This in-progress PR may be relevant to you: https://github.com/opensim-org/opensim-core/pull/2930.

[Pavlos Silvestros]

Afternoon Nick,

Thanks for the quick reply!

Yeah the patch files were in the same folder as the CMakeLists.txt file and I didn't see any errors during the configuration and build regarding them. I can see the MUMPS and METIS header files are installed.

Also gcc (and gfortran) are installed on the system. We are continuing to look into it - it might be an issue with with identifying gfortran over the imkl libraries and defaulting the linear solver to pardiso (?). From any experiences have you needed to set up any specific flags in the configuration or build? The Ipopt INSTALL documentation specifies some flags for using MUMPS instead of pardiso (https://coin-or.github.io/Ipopt/INSTALL.html). Will also try some workarounds during runtime with environment variables.

Thanks again and I will update you if I make any progress.

Cheers,
Pavlos

[Nicholas Bianco]

I haven't needed to set any Ipopt flags to install MUMPS over pardiso in the past, but I also haven't worked much on Linux.

If setting the flags doesn't work, one thing to try is to uninstall pardiso from your cluster machine so that Ipopt doesn't find it before it finds MUMPS.

[Pavlos Silvestros]

Hey Nick,

I think I found a workaround that will work at runtime.

An Ipopt options file (ipopt.opt) can be created in the directory where Ipopt will be executed from that you can specify a lot of the Ipopt options (https://coin-or.github.io/Ipopt/OPTIONS.html) that might not be accessible through the Moco/OpenSim API. I tried it and Ipopt is now using MUMPS and the optimisation is running as I expected (better performance than pardiso).

Hopefully this can be useful if someone wants to tweak some specific Ipopt settings when using Moco?

I will confirm if my test simulations converge as I expected.

Thanks for your help and suggestions again!

Cheers,
Pavlos

[Nicholas Bianco]

Hi Pavlos,

Good to know! That's a handy workaround for now.

If we wanted to set the linear solver through the API, we could add a property to MocoCasADiSolver that updates the Ipopt settings internally (see: https://github.com/opensim-org/opensim- ... r.cpp#L243).

[Pavlos Silvestros]

Hey Nick,

The test simulations I was running on the clusters converged as expected. Using MUMPS the objective and constraint errors for my problems were at least two orders of magnitude smaller on the iteration that the simulation converged compared to the same iteration using pardiso as I mentioned in my first post.

That would be cool if the option existed to select the solver through the API. Have you or anyone else had problems where pardiso (or another available Ipopt linear solver) performed better than MUMPS?

Cheers,
Pavlos

[Nicholas Bianco]

Great! Glad you were able to resolve that.

I don't think I've used any linear solvers outside of MUMPS, so I couldn't say.

[Carlos Gonçalves]

Hello Pavlos. I think your post can help me, but I need guidance...

I'm trying to run my simulation into another computer, very similar to mine, but I keep getting an error regarding MUMPS is out of memory.

The suggestion in the error is to use the configuration mumps_mem_percent config and lower it.

I tried to put it at ipopt.opt file but with no success. Currently the file is in the simulation folder where I call the main python file. Is it the right place?

Best regards.