Hello !
(SimVascular 2021.06.16, Windows 10 21H2)
Unless I'm mistaken, I think there is a bug on MPI with SimVascular for Windows.
With a dual Xeon 5220R (2 x 24 cores, 2 x 48 threads), Simvascular on Windows 10 only sees one of the 2 CPUs. Indeed, when I check the "Use MPI" option in the "Create File and run Simulation" tab, the "Number of processes" slider only goes up to 48 (= number of threads of a single CPU). Moreover, when I run a simulation, only one of the CPUs "heats up".
Under Ubuntu, it is possible to select the 2 CPU (the slider "Number of processes" goes up to 96 )
Thank you in advance for the answer!
fderme
[Bug ?] SimVascular for windows on a dual CPU machine
- François de Kermenguy
- Posts: 13
- Joined: Sat Oct 02, 2021 12:08 am
- David Parker
- Posts: 1705
- Joined: Tue Aug 23, 2005 2:43 pm
Re: [Bug ?] SimVascular for windows on a dual CPU machine
Hi Francois,
SV uses the QThread::idealThreadCount() function to determine the number of cores available for running a simulation using MPI. This function returns the number of physical cores which is 48. By default MPI will use only physical cores and this is what SV supports.
I would guess that a simulation would not benefit from hyperthreading but if you would like to give it a try then you will need to run a simulation from the command line using something like
Cheers,
Dave
SV uses the QThread::idealThreadCount() function to determine the number of cores available for running a simulation using MPI. This function returns the number of physical cores which is 48. By default MPI will use only physical cores and this is what SV supports.
I would guess that a simulation would not benefit from hyperthreading but if you would like to give it a try then you will need to run a simulation from the command line using something like
Code: Select all
mpiexec -n 96 --use-hwthread-cpus svsolver.exe solver.inp
Dave
- François de Kermenguy
- Posts: 13
- Joined: Sat Oct 02, 2021 12:08 am
Re: [Bug ?] SimVascular for windows on a dual CPU machine
Thanks for the answer!
But I still don't understand. For example on my Windows 10 laptop, I have 6 cores / 12 threads, and SimVascular gives the possibility to set "Number of Procesesses" to 12 (even if there is no benefit to use 12 threads, I agree)
And again, on my 2 CPU machine, only 1 of the 2 CPUs is used during the simulation, even if I select 48 processes (in this case, 24 cores + 24 threads of the same CPU are used).
Thanks !
But I still don't understand. For example on my Windows 10 laptop, I have 6 cores / 12 threads, and SimVascular gives the possibility to set "Number of Procesesses" to 12 (even if there is no benefit to use 12 threads, I agree)
And again, on my 2 CPU machine, only 1 of the 2 CPUs is used during the simulation, even if I select 48 processes (in this case, 24 cores + 24 threads of the same CPU are used).
Thanks !
- David Parker
- Posts: 1705
- Joined: Tue Aug 23, 2005 2:43 pm
Re: [Bug ?] SimVascular for windows on a dual CPU machine
Hi Francois,
I spoke too soon without doing a proper test, sorry!
Like you have seen, SV does return the number of logical cores. However, using all of the logical cores will cause MPI to fail, at least on MacOS and Ubuntu if oversubscribe is disabled, does not seem to fail on Windows it seems. I've opened a GitHub Issue for this (see https://github.com/SimVascular/SimVascular/issues/951).
SV simply uses `QThread::idealThreadCount()` to get the number of logical cores, does not know about multiple CPUs, so it will just run the simulation on the CPU that it is running on, should really use `sysctl` to get a better information
To make use of both of your CPUs you will need to run a simulation using MPI from the command line.
Cheers,
Dave
I spoke too soon without doing a proper test, sorry!
Like you have seen, SV does return the number of logical cores. However, using all of the logical cores will cause MPI to fail, at least on MacOS and Ubuntu if oversubscribe is disabled, does not seem to fail on Windows it seems. I've opened a GitHub Issue for this (see https://github.com/SimVascular/SimVascular/issues/951).
SV simply uses `QThread::idealThreadCount()` to get the number of logical cores, does not know about multiple CPUs, so it will just run the simulation on the CPU that it is running on, should really use `sysctl` to get a better information
Code: Select all
$ sysctl -a | grep machdep.cpu | grep core
machdep.cpu.cores_per_package: 8
machdep.cpu.thermal.core_power_limits: 1
machdep.cpu.core_count: 4
Cheers,
Dave
- François de Kermenguy
- Posts: 13
- Joined: Sat Oct 02, 2021 12:08 am
Re: [Bug ?] SimVascular for windows on a dual CPU machine
Thanks for the answer! I will look into it!
For those who are interested, here is the documentation of "mpiexec":
https://docs.microsoft.com/en-us/powers ... w=hpc19-ps
For those who are interested, here is the documentation of "mpiexec":
https://docs.microsoft.com/en-us/powers ... w=hpc19-ps
- François de Kermenguy
- Posts: 13
- Joined: Sat Oct 02, 2021 12:08 am
Re: [Bug ?] SimVascular for windows on a dual CPU machine
Hello,
I have been looking into this and here is the right solution. I contacted the windows MPI team (askmpi@microsoft.com) who answered me very quickly.
My message:
You must add the /a (or /affinity) to use the 2 processors.
I hope it will help some people !
fdekerme
I have been looking into this and here is the right solution. I contacted the windows MPI team (askmpi@microsoft.com) who answered me very quickly.
My message:
Their answer:I need to run a code on my local dual socket PC ( 2 x intel Xeon 24 core/48 thread). I would like to be able to use the 48 physical cores available. However, when I use mpiexec -n 48, 24 cores + 24 threads of one cpu are used. The second cpu is not used. How can I do it?
So, to properly run SimVascular on a dual socket (or multi-node) machine on Windows 10, you need to use the following command lines:This is the common issue with processor groups in Windows. Once your system has more than 64 logical processes, Windows will arrange them into processor groups, where each group has no more than 64 logical processors. Then Windows scheduler will preferentially assign processes to processors from one processor group. To override this behavior, you will have to use the latest msmpi release and -affinity command line option. Please look up the explanation / usage of -affinity and -affinity_layout options from
mpiexec -help2
and pick the layout that is best for your task. I hope it helps.
Code: Select all
cd C:\Users\...\Simulation\simulation_name #the folder where the solver.inp file is located
mpiexec /n [number of cores to use] /a "C:\Program Files\SimVascular\solver2019-05-28\solver-msmpi-bin.exe" "C:\Users\...\Simulation\simulation_name\solver.inp
I hope it will help some people !
fdekerme
- David Parker
- Posts: 1705
- Joined: Tue Aug 23, 2005 2:43 pm
Re: [Bug ?] SimVascular for windows on a dual CPU machine
Hi fdekerme,
This is very helpful to me and I'm sure to other users also, thanks for posting this!
I will add this to the SV documentation.
Cheers,
Dave
This is very helpful to me and I'm sure to other users also, thanks for posting this!
I will add this to the SV documentation.
Cheers,
Dave