Running svSolver in Cluster

[Damian Deza]

Hello, in the past few weeks i tried running a simulation directly with svSolver in a CentOS Linux release 7.6.1810 (Core) cluster having SLURM for work queues.
Everytime i try to run a job i get this output

/home/*****/svsolver/2019-02-07/bin/svsolver: symbol lookup error: /home/******/svsolver/2019-02-07/bin/svsolver: undefined symbol: mpi_type_extent_
srun: error: nodo13: tasks 0-23: Exited with exit code 127

The cluster has mpich-3.2.1.

I've tryed running the same simulation with mpich-2.1.5, but i get another error: undefined symbol: mpi_comm_rank_

Am i missing any steps?

Thank you.

[David Parker]

Hi Damian,

The svSolver installer is for Ubuntu so I am assuming that you build the CentOS version yourself?

Cheers,
Dave

[Damian Deza]

David,
I download the Linux version from the download page then proceeded to unpack it from the .deb
Never did any builds.

The weird thing is that i was able to run the solver in a VM with centOS and installing mpi with the command

yum install mpich

I'm only having issues with the cluster.

I'm executing the solver in this way:

*cd on the pre solver created folder*
mpiexec -n 24 /home/*****/svsolver/2019-02-07/svSolver

tryed as well like this
mpiexec -n 24 /home/*****/svsolver/2019-02-07/bin/svSolver

[David Parker]

Hi Damian,

Hmmm, I would not think that this would work, but maybe with the right CentOS it could.

I'm not sure what is causing the undefined symbol: mpi_type_extent error, seems like the library you are linking to can't find this symbol, maybe due to CentOS / Ubuntu. Are you sure you are using an mpich library and not OpenMPI? Maybe check your LD_LIBRARY env variable to see.

Cheers,
Dave

[Damian Deza]

David,

I was able to get the mpi version with the command "mpicc -v", this is the one currently on the LD_LIBRARY_PATH and i'm getting this output:

Code: Select all

mpicc for MPICH version 3.2.1
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)

I'm going to try to execute with MPICH2 to see if i get a different output on the script. This is the output i get on the mpicc -v for mpich2:

Code: Select all

mpicc for MPICH2 version 1.5
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)

I'll tell you what i get after the script is executed on the cluster (there are currently a few jobs on queue haha).

Thank you!

[David Parker]

Hi Damian,

Well, I created a VM with CentOS 7 and ran svsolver no problem with both MPICH and OpenMPI! I didn't think that would work ...

The problem must have to do with setting up the environment you are running the job in to find MPI. I know on some clusters you must manually ask for certain applications to be loaded before you execute a job.

Cheers,
Dave