Page 1 of 1
Simulation Computer Specs Recommendations?
Posted: Mon May 02, 2022 6:35 am
by eadooley
Hello all,
I was wondering if any of you all had recommendations for building a machine to run simulations in Moco. I have been running torque-driven simulations so far and not run into any issues with my existing machine, but in setting up muscle-driven simulations I am maxing out the memory on my computer.
I am currently running i5 CPU at 3.2GHz with 8 Gb RAM, but recognize that will not be enough memory. If I shorten the length of the motion I am trying to simulate I can get the problem to initialize, but then the memory caps out as it tries to iterate. I took my set-up over to a lab with more powerful computers (i7 CPU with 32 Gb RAM) and the study does converge (so that seems promising).
I saw in the Moco paper that the time comparisons are given for a machine with a "3.6 GHz Intel Core i7 processor with 8 parallel threads", additionally I saw Dr. Miller was running simulations on a machine with "3.8-GHz 10-core Intel Core i9 CPU." Then there is this thread suggesting 8 cores as the optimal number to run a Moco simulation with (
viewtopicPhpbb.php?f=1815&t=13391&p=384 ... 2cc244e0c2)
The discussion seems to come down to speed is aided by faster processor, and doesn't greatly improve above 8-10 cores for a simulation. I haven't found such discussion for how memory is effected. Do any of you have recommendations about how much memory works well for your simulations?
If we are to build a simulation machine in my lab I wanted to make sure that I am setting us up for success with these larger muscle-driven simulations. Any thoughts you all have about the best machines for running these simulations would be appreciated.
Thanks,
Evan
Re: Simulation Computer Specs Recommendations?
Posted: Mon May 02, 2022 8:01 am
by rosshm
Hi Evan,
Brian's data if I remember right showed diminishing returns beyond 10 cores, but still was getting speed improvements up to 20+ cores I think. Maybe Brian will see this and clarify. Personally if I was building a Moco/IPOPT/simulations machine, I would get as many cores as I could afford.
My desktop and laptop both have 64 GB of RAM and I've not yet run into any memory issues with Moco. The largest model I've used is 31 DoF and 120 muscles over a full stride of walking with 200 collocation mesh intervals.
If you are planning getting a Mac, I think there is not yet a native version of Matlab for the Apple Silicon "M1". I've heard Intel-native Matlab still runs fine on it but don't know of anyone who has tried Moco on it. One of our students actually just got a new M1 Mac and is going to try Matlab/Moco, I can report back on her results. I would assume no issues with Python/C++ flavors of Moco on M1 but also have no firsthand experience with that.
I keep an eye on the MacRumors Buyer's Guide when I'm thinking of getting something. The problem with Apple is there's always a bunch of rumors on how awesome the next machine is going to be, makes me want to never buy the current machine!
https://buyersguide.macrumors.com/#mac
Ross
Re: Simulation Computer Specs Recommendations?
Posted: Mon May 02, 2022 10:00 am
by eadooley
Thanks. That's great to hear about the complexity of your model. I am only up to 80 muscles so far.
My PI would like to stay away from Macs, so most likely we would be running on an Intel chip anyway. It would be great to hear if the M1 runs Matlab/Moco though. It does seem to very good at lots of other things.
Thanks,
Evan
Re: Simulation Computer Specs Recommendations?
Posted: Mon May 02, 2022 2:19 pm
by pvb
Hi Evan,
I'm a bit surprised that you're having RAM issues - my (10 year old) i5 desktop with 4GB ram essentially runs moco fine (but just incredibly slowly). This was up to 14 degrees of freedom with 22 muscles. The PC I built to replace it (with the Ryzen 5900x 12 core/24 thread processor) initially had 32GB, now it's got 64GB, but I don't think the RAM has affected my simulations at all.
Since that thread, I had a conversation with a computer scientist who told me that you usually won't see much of a computational benefit from using anything beyond the actual physical cores you have (e.g. I have 12 cores, but 24 threads). My plots in the thread do support that interpretation, although it might be hard to tell because of the diminishing returns after 10 cores found by Brian Umberger('s PhD student).
If you have plans on trying shooting methods as well (e.g. SCONE), you'll definitely benefit from all the cores you can afford. The same is true for Moco, especially if you might want to run several optimizations in parallel.
Cheers,
Pasha
Re: Simulation Computer Specs Recommendations?
Posted: Tue May 03, 2022 7:17 am
by rosshm
RAM I don't think will have much influence on how fast IPOPT iterates, there just needs to be enough memory for the program to run. I'm no computer engineer though.
Evan, are you using Matlab? I previously ran into the memory issue you described (not enough memory to initialize/run the problem) when using IPOPT on some larger custom DC problems some years ago. Matlab also had a notorious memory leak for a while although I think it was resolved several versions ago.
Ross
Re: Simulation Computer Specs Recommendations?
Posted: Fri Apr 14, 2023 6:05 am
by carlosoleto
Hello everyone,
@Evan, can you confirm that your initial error was something like this below?
Code: Select all
This is Ipopt version 3.12.8, running with linear solver mumps.
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).
Number of nonzeros in equality constraint Jacobian...: 48902396
Number of nonzeros in inequality constraint Jacobian.: 0
Number of nonzeros in Lagrangian Hessian.............: 0
MUMPS returned INFO(1) =-13 - out of memory when trying to allocate 1462133103 bytes.
In some cases it helps to decrease the value of the option "mumps_mem_percent".
Total number of variables............................: 81575
variables with only lower bounds: 0
variables with lower and upper bounds: 81575
variables with only upper bounds: 0
Total number of equality constraints.................: 72184
Total number of inequality constraints...............: 0
inequality constraints with only lower bounds: 0
inequality constraints with lower and upper bounds: 0
inequality constraints with only upper bounds: 0
Warning: intermediate_callback is disfunctional in your installation. You will only be able to use stats(). See https://github.com/casadi/casadi/wiki/enableIpoptCallback to enable it.
iter objective inf_pr inf_du lg(mu) ||d|| lg(rg) alpha_du alpha_pr ls
0 1.4036351e+02 1.67e+01 1.05e+02 0.0 0.00e+00 - 0.00e+00 0.00e+00 0
MUMPS returned INFO(1) =-13 - out of memory when trying to allocate 1462133103 bytes.
In some cases it helps to decrease the value of the option "mumps_mem_percent".
WARNING: Problem in step computation; switching to emergency mode.
I'm trying to run my simulations on a virtual machine with 8 threads and 14GB of RAM and keep getting this error. It is strange because, as Pasha commented, I had it running at other less powerful machines before. And the same simulation is running right now at my laptop (16 threads, 16GB).
Any advice?
Best regards.
Re: Simulation Computer Specs Recommendations?
Posted: Fri Apr 14, 2023 7:41 am
by carlosoleto
Adding on my previous message, I found this reference in IPOPT page
https://coin-or.github.io/Ipopt/OPTIONS ... ear_Solver
mumps_mem_percent: Percentage increase in the estimated working space for MUMPS.
When significant extra fill-in is caused by numerical pivoting, larger values of mumps_mem_percent may help use the workspace more efficiently. On the other hand, if memory requirement are too large at the very beginning of the optimization, choosing a much smaller value for this option, such as 5, might reduce memory requirements. The valid range for this integer option is 0 ,<= mumps_mem_percent and its default value is 1000.
Is there a way tho change this parameter before running study.solve() ?
Best regards.