[Bug ?] SimVascular for windows on a dual CPU machine

[François de Kermenguy]

Hello !

(SimVascular 2021.06.16, Windows 10 21H2)

Unless I'm mistaken, I think there is a bug on MPI with SimVascular for Windows.

With a dual Xeon 5220R (2 x 24 cores, 2 x 48 threads), Simvascular on Windows 10 only sees one of the 2 CPUs. Indeed, when I check the "Use MPI" option in the "Create File and run Simulation" tab, the "Number of processes" slider only goes up to 48 (= number of threads of a single CPU). Moreover, when I run a simulation, only one of the CPUs "heats up".

Under Ubuntu, it is possible to select the 2 CPU (the slider "Number of processes" goes up to 96 )

Thank you in advance for the answer!
fderme

[David Parker]

Hi Francois,

SV uses the QThread::idealThreadCount() function to determine the number of cores available for running a simulation using MPI. This function returns the number of physical cores which is 48. By default MPI will use only physical cores and this is what SV supports.

I would guess that a simulation would not benefit from hyperthreading but if you would like to give it a try then you will need to run a simulation from the command line using something like

Code: Select all

mpiexec -n 96 --use-hwthread-cpus svsolver.exe solver.inp

Cheers,
Dave

[François de Kermenguy]

Thanks for the answer!
But I still don't understand. For example on my Windows 10 laptop, I have 6 cores / 12 threads, and SimVascular gives the possibility to set "Number of Procesesses" to 12 (even if there is no benefit to use 12 threads, I agree)

And again, on my 2 CPU machine, only 1 of the 2 CPUs is used during the simulation, even if I select 48 processes (in this case, 24 cores + 24 threads of the same CPU are used).

CPU_usage_2.PNG (29.43 KiB) Viewed 379 times

CPU_usage.PNG (13.17 KiB) Viewed 379 times

CPU_usage_3.PNG (6.89 KiB) Viewed 379 times

Thanks !

[David Parker]

Hi Francois,

I spoke too soon without doing a proper test, sorry!

Like you have seen, SV does return the number of logical cores. However, using all of the logical cores will cause MPI to fail, at least on MacOS and Ubuntu if oversubscribe is disabled, does not seem to fail on Windows it seems. I've opened a GitHub Issue for this (see https://github.com/SimVascular/SimVascular/issues/951).

SV simply uses `QThread::idealThreadCount()` to get the number of logical cores, does not know about multiple CPUs, so it will just run the simulation on the CPU that it is running on, should really use `sysctl` to get a better information

Code: Select all

$ sysctl -a | grep machdep.cpu | grep core
machdep.cpu.cores_per_package: 8
machdep.cpu.thermal.core_power_limits: 1
machdep.cpu.core_count: 4

To make use of both of your CPUs you will need to run a simulation using MPI from the command line.

Cheers,
Dave

[François de Kermenguy]

Thanks for the answer! I will look into it!
For those who are interested, here is the documentation of "mpiexec":
https://docs.microsoft.com/en-us/powers ... w=hpc19-ps

[François de Kermenguy]

Hello,
I have been looking into this and here is the right solution. I contacted the windows MPI team (askmpi@microsoft.com) who answered me very quickly.
My message:

I need to run a code on my local dual socket PC ( 2 x intel Xeon 24 core/48 thread). I would like to be able to use the 48 physical cores available. However, when I use mpiexec -n 48, 24 cores + 24 threads of one cpu are used. The second cpu is not used. How can I do it?

Their answer:

This is the common issue with processor groups in Windows. Once your system has more than 64 logical processes, Windows will arrange them into processor groups, where each group has no more than 64 logical processors. Then Windows scheduler will preferentially assign processes to processors from one processor group. To override this behavior, you will have to use the latest msmpi release and -affinity command line option. Please look up the explanation / usage of -affinity and -affinity_layout options from

mpiexec -help2

and pick the layout that is best for your task. I hope it helps.

So, to properly run SimVascular on a dual socket (or multi-node) machine on Windows 10, you need to use the following command lines:

Code: Select all

cd C:\Users\...\Simulation\simulation_name #the folder where the solver.inp file is located

mpiexec /n [number of cores to use] /a  "C:\Program Files\SimVascular\solver2019-05-28\solver-msmpi-bin.exe" "C:\Users\...\Simulation\simulation_name\solver.inp

You must add the /a (or /affinity) to use the 2 processors.

I hope it will help some people !

fdekerme

[David Parker]

Hi fdekerme,

This is very helpful to me and I'm sure to other users also, thanks for posting this!

I will add this to the SV documentation.

Cheers,
Dave