Get Ipopt to use mumps vs pardiso
- Pavlos Silvestros
- Posts: 13
- Joined: Tue Apr 12, 2022 9:57 am
Get Ipopt to use mumps vs pardiso
Hello everyone,
With some help I have managed to get OpenSim Moco running on some clusters I have access to. In my tests on the cluster (built from source) I get my optimal control problems to run however I have noticed that Ipopt is using pardiso. This is different to my local machines (not built from source) that use mumps and also the function error when using pardiso is at least 100x larger for the same sims when using pardiso vs mumps.
I have tested this on both Intel and AMD processors, and tried loading specific mumps-metis libraries after installation as a last ditch attempt but still encountering it. Is there a way to specify mumps during runtime or is there a flag that I missed during compiling/building?
As I am writing this I realised that my local machines are running OpenSim 4.3 whereas I installed 4.4.... Before I do everything again, would you expect that to have an effect due to an update?
I see they are a couple more threads related to mumps and HPCs but couldn't quite understand to to apply the answers to my problem.
Thanks as always!
Cheers,
Pavlos
--Some info on the cluster system
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.80.1.el7.x86_64
Architecture: x86-64
With some help I have managed to get OpenSim Moco running on some clusters I have access to. In my tests on the cluster (built from source) I get my optimal control problems to run however I have noticed that Ipopt is using pardiso. This is different to my local machines (not built from source) that use mumps and also the function error when using pardiso is at least 100x larger for the same sims when using pardiso vs mumps.
I have tested this on both Intel and AMD processors, and tried loading specific mumps-metis libraries after installation as a last ditch attempt but still encountering it. Is there a way to specify mumps during runtime or is there a flag that I missed during compiling/building?
As I am writing this I realised that my local machines are running OpenSim 4.3 whereas I installed 4.4.... Before I do everything again, would you expect that to have an effect due to an update?
I see they are a couple more threads related to mumps and HPCs but couldn't quite understand to to apply the answers to my problem.
Thanks as always!
Cheers,
Pavlos
--Some info on the cluster system
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.80.1.el7.x86_64
Architecture: x86-64
- Nicholas Bianco
- Posts: 1027
- Joined: Thu Oct 04, 2012 8:09 pm
Re: Get Ipopt to use mumps vs pardiso
Hi Pavlos,
Here's the CMake for building Ipopt on Unix machines: https://github.com/opensim-org/opensim- ... s.txt#L318.
The patch file for getting MUMPS is located in the same directory in the source.
MUMPS requires gfortran. Have you double checked that you're installing it (brew install gcc) on your cluster machine?
Best,
Nick
Here's the CMake for building Ipopt on Unix machines: https://github.com/opensim-org/opensim- ... s.txt#L318.
The patch file for getting MUMPS is located in the same directory in the source.
MUMPS requires gfortran. Have you double checked that you're installing it (brew install gcc) on your cluster machine?
Best,
Nick
- Nicholas Bianco
- Posts: 1027
- Joined: Thu Oct 04, 2012 8:09 pm
Re: Get Ipopt to use mumps vs pardiso
This in-progress PR may be relevant to you: https://github.com/opensim-org/opensim-core/pull/2930.
- Pavlos Silvestros
- Posts: 13
- Joined: Tue Apr 12, 2022 9:57 am
Re: Get Ipopt to use mumps vs pardiso
Afternoon Nick,
Thanks for the quick reply!
Yeah the patch files were in the same folder as the CMakeLists.txt file and I didn't see any errors during the configuration and build regarding them. I can see the MUMPS and METIS header files are installed.
Also gcc (and gfortran) are installed on the system. We are continuing to look into it - it might be an issue with with identifying gfortran over the imkl libraries and defaulting the linear solver to pardiso (?). From any experiences have you needed to set up any specific flags in the configuration or build? The Ipopt INSTALL documentation specifies some flags for using MUMPS instead of pardiso (https://coin-or.github.io/Ipopt/INSTALL.html). Will also try some workarounds during runtime with environment variables.
Thanks again and I will update you if I make any progress.
Cheers,
Pavlos
Thanks for the quick reply!
Yeah the patch files were in the same folder as the CMakeLists.txt file and I didn't see any errors during the configuration and build regarding them. I can see the MUMPS and METIS header files are installed.
Also gcc (and gfortran) are installed on the system. We are continuing to look into it - it might be an issue with with identifying gfortran over the imkl libraries and defaulting the linear solver to pardiso (?). From any experiences have you needed to set up any specific flags in the configuration or build? The Ipopt INSTALL documentation specifies some flags for using MUMPS instead of pardiso (https://coin-or.github.io/Ipopt/INSTALL.html). Will also try some workarounds during runtime with environment variables.
Thanks again and I will update you if I make any progress.
Cheers,
Pavlos
- Nicholas Bianco
- Posts: 1027
- Joined: Thu Oct 04, 2012 8:09 pm
Re: Get Ipopt to use mumps vs pardiso
I haven't needed to set any Ipopt flags to install MUMPS over pardiso in the past, but I also haven't worked much on Linux.
If setting the flags doesn't work, one thing to try is to uninstall pardiso from your cluster machine so that Ipopt doesn't find it before it finds MUMPS.
If setting the flags doesn't work, one thing to try is to uninstall pardiso from your cluster machine so that Ipopt doesn't find it before it finds MUMPS.
- Pavlos Silvestros
- Posts: 13
- Joined: Tue Apr 12, 2022 9:57 am
Re: Get Ipopt to use mumps vs pardiso
Hey Nick,
I think I found a workaround that will work at runtime.
An Ipopt options file (ipopt.opt) can be created in the directory where Ipopt will be executed from that you can specify a lot of the Ipopt options (https://coin-or.github.io/Ipopt/OPTIONS.html) that might not be accessible through the Moco/OpenSim API. I tried it and Ipopt is now using MUMPS and the optimisation is running as I expected (better performance than pardiso).
Hopefully this can be useful if someone wants to tweak some specific Ipopt settings when using Moco?
I will confirm if my test simulations converge as I expected.
Thanks for your help and suggestions again!
Cheers,
Pavlos
I think I found a workaround that will work at runtime.
An Ipopt options file (ipopt.opt) can be created in the directory where Ipopt will be executed from that you can specify a lot of the Ipopt options (https://coin-or.github.io/Ipopt/OPTIONS.html) that might not be accessible through the Moco/OpenSim API. I tried it and Ipopt is now using MUMPS and the optimisation is running as I expected (better performance than pardiso).
Hopefully this can be useful if someone wants to tweak some specific Ipopt settings when using Moco?
I will confirm if my test simulations converge as I expected.
Thanks for your help and suggestions again!
Cheers,
Pavlos
- Nicholas Bianco
- Posts: 1027
- Joined: Thu Oct 04, 2012 8:09 pm
Re: Get Ipopt to use mumps vs pardiso
Hi Pavlos,
Good to know! That's a handy workaround for now.
If we wanted to set the linear solver through the API, we could add a property to MocoCasADiSolver that updates the Ipopt settings internally (see: https://github.com/opensim-org/opensim- ... r.cpp#L243).
Good to know! That's a handy workaround for now.
If we wanted to set the linear solver through the API, we could add a property to MocoCasADiSolver that updates the Ipopt settings internally (see: https://github.com/opensim-org/opensim- ... r.cpp#L243).
- Pavlos Silvestros
- Posts: 13
- Joined: Tue Apr 12, 2022 9:57 am
Re: Get Ipopt to use mumps vs pardiso
Hey Nick,
The test simulations I was running on the clusters converged as expected. Using MUMPS the objective and constraint errors for my problems were at least two orders of magnitude smaller on the iteration that the simulation converged compared to the same iteration using pardiso as I mentioned in my first post.
That would be cool if the option existed to select the solver through the API. Have you or anyone else had problems where pardiso (or another available Ipopt linear solver) performed better than MUMPS?
Cheers,
Pavlos
The test simulations I was running on the clusters converged as expected. Using MUMPS the objective and constraint errors for my problems were at least two orders of magnitude smaller on the iteration that the simulation converged compared to the same iteration using pardiso as I mentioned in my first post.
That would be cool if the option existed to select the solver through the API. Have you or anyone else had problems where pardiso (or another available Ipopt linear solver) performed better than MUMPS?
Cheers,
Pavlos
- Nicholas Bianco
- Posts: 1027
- Joined: Thu Oct 04, 2012 8:09 pm
Re: Get Ipopt to use mumps vs pardiso
Great! Glad you were able to resolve that.
I don't think I've used any linear solvers outside of MUMPS, so I couldn't say.
I don't think I've used any linear solvers outside of MUMPS, so I couldn't say.
- Carlos Gonçalves
- Posts: 135
- Joined: Wed Jun 08, 2016 4:56 am
Re: Get Ipopt to use mumps vs pardiso
Hello Pavlos. I think your post can help me, but I need guidance...
I'm trying to run my simulation into another computer, very similar to mine, but I keep getting an error regarding MUMPS is out of memory.
The suggestion in the error is to use the configuration mumps_mem_percent config and lower it.
I tried to put it at ipopt.opt file but with no success. Currently the file is in the simulation folder where I call the main python file. Is it the right place?
Best regards.
I'm trying to run my simulation into another computer, very similar to mine, but I keep getting an error regarding MUMPS is out of memory.
The suggestion in the error is to use the configuration mumps_mem_percent config and lower it.
I tried to put it at ipopt.opt file but with no success. Currently the file is in the simulation folder where I call the main python file. Is it the right place?
Best regards.