Running GenBC in parallel

[Luca Pegolotti]

Hi,

I need to run some coronary simulations with prescribed pressure at the inlet. To do so, I implemented the usual coronary bcs + prescribed pressure at the inlet in GenBC. In serial this seems to work fine, although it's a bit difficult to understand if my implementation is correct.

When I try to run the simulation in parallel with mpirun -n (>= 2), I get weird errors like the following

At line 78 of file USER.f (unit = 1)
Fortran runtime error: Cannot open file 'InitialData': File exists

or

ERROR: Number of Dirichlet Surfaces from Phasta is: 388685675
While nDirichletSrfs is equal to: 0 <- this is correct as all the faces are Neumann

(when run twice).

Considering that these errors occur at lines that I did not modify from the provided USER.f, I am wondering if svsolver fully supports parallel computations when GenBC is used.

Luca

[Luca Pegolotti]

Some additional information: I am using Mac OS 11.3 and the errors were also reproduced on a Linux cluster. Version of gfortan: 10.2.0. Version of svsolver: latest. Unfortunately I couldn't attach the simulation files even with a very coarse mesh due to limitations in the size of the attachments. Attached you find the custom GenBC I am using (note that I added the possibility to pass time-varying inlet and myocardial pressures).

[David Parker]

Hi Luca,

I've reproduced this problem on my MacOS 10.13 using /usr/local/sv/svsolver/2019-01-19/svsolver. This uses MPICH libraries in /usr/local/sv/svsolver/2019-01-19/lib.

I can execute ok using mpi using an svSolver version compiled from source using OpenMPI installed on my Mac.

Do you have OpenMPI or MPICH installed on your Mac? I'm thinking the problem is an mpi issue.

Cheers,
Dave

[Luca Pegolotti]

Hi Dave,

Thanks for taking the time to look into this. I am using the built-in binaries for Mac. I couldn't install svsolver using a custom installation of OpenMPI because my installation (weirdly enough) does not have any .cmake files I can add to my path to let the FinMPI script within svsolver find it. Simply bypassing FindMPI by adding the path of my includes and libs causes a cmake error. What is the correct way to install svsolver with a custom mpi?

By the way, the error I had on the cluster was due to a bug in my GenBC rather than the mpi version. There I can use mvapich2 without any issues.

Cheers,
Luca

[David Parker]

Hi Luca,

I'm glad you found the problem! Getting GenBC to work correctly is indeed tricky.

I can build svSolver fine on my MacOS 10.13 with OpenMPI installed like this

Code: Select all

git clone https://github.com/SimVascular/svSolver.git
cd svSolver
mkdir build
cd build
cmake ..
make

There might be an issue with Mac OS 11.3, which CMake version are you using? Also, which GCC version are you running?

Cheers,
Dave