Installation problems with CUDA-2.2 OpenMM

[Siddharth Srinivasan]

Hi Sherm

No its pretty consistent across all the tests, HelloSodiumChloride also fails during the cleanup portion

[Michael Sherman]

If you can run that one in gdb and get a stack trace it might be more informative than HelloArgon where the cleanup is being done inside anonymous destructor methods. We should be able to tell exactly what is being destroyed when disaster strikes.

[Siddharth Srinivasan]

I'm not sure this will be any more illuminating, but
{{{
REMARK 250 time=9.950 ps; energy=-412.080 kcal/mole
ATOM 1 NA SLT 1 3.052 -5.476 0.147 1.00 0.00
ATOM 2 CL SLT 1 -3.912 0.512 -4.169 1.00 0.00
ATOM 3 NA SLT 1 -0.235 6.988 3.051 1.00 0.00
ATOM 4 CL SLT 1 1.857 -5.465 -2.025 1.00 0.00
ATOM 5 NA SLT 1 -4.354 -0.412 -6.523 1.00 0.00
ATOM 6 CL SLT 1 2.002 0.309 9.239 1.00 0.00
ENDMDL
*** glibc detected *** /Network/Cluster/home/siddharth/Downloads/openmmExamples/HelloSodiumChloride: double free or corruption (fasttop): 0x000000000050acc0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7fb8734c354b]
/lib/libc.so.6(__libc_free+0x8c)[0x7fb8734c6c7c]
/Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so[0x7fb873b7cb91]
/lib/libc.so.6(__cxa_finalize+0x9d)[0x7fb873485a4d]
/Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so[0x7fb873b5a5d3]
======= Memory map: ========
00400000-00407000 r-xp 00000000 00:22 3489873848 /Network/Cluster/home/siddharth/Downloads/openmmExamples/HelloSodiumChloride
00507000-00508000 rw-p 00007000 00:22 3489873848 /Network/Cluster/home/siddharth/Downloads/openmmExamples/HelloSodiumChloride
00508000-01cac000 rw-p 00508000 00:00 0 [heap]
7fb868000000-7fb868021000 rw-p 7fb868000000 00:00 0
7fb868021000-7fb86c000000 ---p 7fb868021000 00:00 0
7fb86f446000-7fb86f546000 rw-s 3f018e000 00:1b 2878 /dev/nvidia0
7fb86f546000-7fb86f646000 rw-s 42f8a0000 00:1b 2878 /dev/nvidia0
7fb86f646000-7fb86f746000 rw-s 3f040b000 00:1b 2878 /dev/nvidia0
7fb86f746000-7fb86f846000 rw-s 42f9c6000 00:1b 2878 /dev/nvidia0
7fb86f846000-7fb86f847000 rw-s 42f9c2000 00:1b 2878 /dev/nvidia0
7fb86f847000-7fb86f848000 rw-s d9c04000 00:1b 2878 /dev/nvidia0
7fb86f848000-7fb86f849000 rw-s 42f9c1000 00:1b 2878 /dev/nvidia0
7fb86f849000-7fb86fc4b000 rw-s 42fdcc000 00:1b 2878 /dev/nvidia0
7fb86fc4b000-7fb87004d000 rw-s 42f4e0000 00:1b 2878 /dev/nvidia0
7fb87004d000-7fb8712eb000 r-xp 00000000 00:22 3760723877 /Network/Cluster/home/siddharth/OpenMM3/lib/plugins/libOpenMMCuda.so
7fb8712eb000-7fb8713eb000 ---p 0129e000 00:22 3760723877 /Network/Cluster/home/siddharth/OpenMM3/lib/plugins/libOpenMMCuda.so
7fb8713eb000-7fb8713ef000 rw-p 0129e000 00:22 3760723877 /Network/Cluster/home/siddharth/OpenMM3/lib/plugins/libOpenMMCuda.so
7fb8713ef000-7fb8713f4000 rw-p 7fb8713ef000 00:00 0
7fb8719d5000-7fb8719dd000 r-xp 00000000 00:0e 2077892 /lib64/librt-2.6.1.so
7fb8719dd000-7fb871adc000 ---p 00008000 00:0e 2077892 /lib64/librt-2.6.1.so
7fb871adc000-7fb871ade000 rw-p 00007000 00:0e 2077892 /lib64/librt-2.6.1.so
7fb871ade000-7fb871af2000 r-xp 00000000 00:0e 2077887 /lib64/libpthread-2.6.1.so
7fb871af2000-7fb871bf2000 ---p 00014000 00:0e 2077887 /lib64/libpthread-2.6.1.so
7fb871bf2000-7fb871bf4000 rw-p 00014000 00:0e 2077887 /lib64/libpthread-2.6.1.so
7fb871bf4000-7fb871bf8000 rw-p 7fb871bf4000 00:00 0
7fb871bf8000-7fb871d0a000 r-xp 00000000 00:22 25291 /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM_d.so
7fb871d0a000-7fb871e0a000 ---p 00112000 00:22 25291 /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM_d.so
7fb871e0a000-7fb871e13000 rw-p 00112000 00:22 25291 /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM_d.so
7fb871e13000-7fb871e14000 rw-p 7fb871e13000 00:00 0
7fb871e14000-7fb871e50000 r-xp 00000000 00:0e 575127 /opt/cuda/lib/libcudart.so.2.2
7fb871e50000-7fb871f4f000 ---p 0003c000 00:0e 575127 /opt/cuda/lib/libcudart.so.2.2
7fb871f4f000-7fb871f51000 rw-p 0003b000 00:0e 575127 /opt/cuda/lib/libcudart.so.2.2
7fb871f51000-7fb873243000 r-xp 00000000 00:22 3760723887 /Network/Cluster/home/siddharth/OpenMM3/lib/plugins/libOpenMMCuda_d.so
7fb873243000-7fb873342000 ---p 012f2000 00:22 3760723887 /Network/Cluster/home/siddharth/OpenMM3/lib/plugins/libOpenMMCuda_d.so
7fb873342000-7fb873349000 rw-p 012f1000 00:22 3760723887 /Network/Cluster/home/siddharth/OpenMM3/lib/plugins/libOpenMMCuda_d.so
7fb873349000-7fb87334e000 rw-p 7fb873349000 00:00 0
7fb87334e000-7fb873351000 r-xp 00000000 00:0e 2077885 /lib64/libdl-2.6.1.so
7fb873351000-7fb873450000 ---p 00003000 00:0e 2077885 /lib64/libdl-2.6.1.so
7fb873450000-7fb873452000 rw-p 00002000 00:0e 2077885 /lib64/libdl-2.6.1.so
7fb873452000-7fb8735a0000 r-xp 00000000 00:0e 2077901 /lib64/libc-2.6.1.so
7fb8735a0000-7fb87369f000 ---p 0014e000 00:0e 2077901 /lib64/libc-2.6.1.so
7fb87369f000-7fb8736a2000 r--p 0014d000 00:0e 2077901 /lib64/libc-2.6.1.so
7fb8736a2000-7fb8736a4000 rw-p 00150000 00:0e 2077901 /lib64/libc-2.6.1.so
7fb8736a4000-7fb8736a9000 rw-p 7fb8736a4000 00:00 0
7fb8736a9000-7fb8736b5000 r-xp 00000000 00:0e 2077779 /lib64/libgcc_s.so.1
7fb8736b5000-7fb8737b5000 ---p 0000c000 00:0e 2077779 /lib64/libgcc_s.so.1
7fb8737b5000-7fb8737b6000 rw-p 0000c000 00:0e 2077779 /lib64/libgcc_s.so.1
7fb8737b6000-7fb873837000 r-xp 00000000 00:0e 2077677 /lib64/libm-2.6.1.so
7fb873837000-7fb873936000 ---p 00081000 00:0e 2077677 /lib64/libm-2.6.1.so
7fb873936000-7fb873938000 rw-p 00080000 00:0e 2077677 /lib64/libm-2.6.1.so
7fb873938000-7fb873a1b000 r-xp 00000000 00:0e 2151807443 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6.0.8
7fb873a1b000-7fb873b1b000 ---p 000e3000 00:0e 2151807443 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6.0.8
7fb873b1b000-7fb873b21000 r--p 000e3000 00:0e 2151807443 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6.0.8
7fb873b21000-7fb873b24000 rw-p 000e9000 00:0e 2151807443 /usr/lib64/gcc/x86_64-pc-linux-gnu/4.1.2/libstdc++.so.6.0.8
7fb873b24000-7fb873b36000 rw-p 7fb873b24000 00:00 0
7fb873b36000-7fb873bbb000 r-xp 00000000 00:22 24657 /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
7fb873bbb000-7fb873cba000 ---p 00085000 00:22 24657 /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
7fb873cba000-7fb873cbe000 rw-p 00084000 00:22 24657 /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
7fb873cbe000-7fb873cc0000 rw-p 7fb873cbe000 00:00 0
7fb873cc0000-7fb873cdd000 r-xp 00000000 00:0e 2077900 /lib64/ld-2.6.1.so
7fb873dbd000-7fb873dc1000 rw-p 7fb873dbd000 00:00 0
7fb873dc6000-7fb873dc7000 rw-p 7fb873dc6000 00:00 0
7fb873dc7000-7fb873dc8000 rw-s d9c02000 00:1b 2878 /dev/nvidia0
7fb873dc8000-7fb873dc9000 rw-s 42f5a3000 00:1b 2878 /dev/nvidia0
7fb873dc9000-7fb873dda000 rw-s 3f048e000 00:1b 2878 /dev/nvidia0
7fb873dda000-7fb873ddc000 rw-p 7fb873dda000 00:00 0
7fb873ddc000-7fb873ddd000 r--p 0001c000 00:0e 2077900 /lib64/ld-2.6.1.so
7fb873ddd000-7fb873dde000 rw-p 0001d000 00:0e 2077900 /lib64/ld-2.6.1.so
7fff7bd00000-7fff7bdde000 rw-p 7ffffff21000 00:00 0 [stack]
7fff7bdfe000-7fff7bdff000 r-xp 7fff7bdfe000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

Program received signal SIGABRT, Aborted.
0x00007fb873482b05 in raise () from /lib/libc.so.6
(gdb) list
106 atoms[n].posInAng[0], atoms[n].posInAng[1], atoms[n].posInAng[2]);
107 printf("ENDMDL\n");
108 }
109
110
111 // -----------------------------------------------------------------------------
112 // MAIN PROGRAM
113 // -----------------------------------------------------------------------------
114 int main() {
115 const int NumReports = (int)(SimulationTimeInPs*1000 / ReportIntervalInFs + 0.5);
(gdb) bt
#0 0x00007fb873482b05 in raise () from /lib/libc.so.6
#1 0x00007fb8734843be in abort () from /lib/libc.so.6
#2 0x00007fb8734bbc97 in ?? () from /lib/libc.so.6
#3 0x00007fb8734c354b in ?? () from /lib/libc.so.6
#4 0x00007fb8734c6c7c in free () from /lib/libc.so.6
#5 0x00007fb873b7cb91 in __tcf_3 () from /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
#6 0x00007fb873485a4d in __cxa_finalize () from /lib/libc.so.6
#7 0x00007fb873b5a5d3 in __do_global_dtors_aux () from /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
#8 0x00007fff7bddc8e0 in ?? ()
#9 0x00007fb873ba6d01 in _fini () from /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
#10 0x0000000000000000 in ?? ()
(gdb) up
#1 0x00007fb8734843be in abort () from /lib/libc.so.6
(gdb)
#2 0x00007fb8734bbc97 in ?? () from /lib/libc.so.6
(gdb)
#3 0x00007fb8734c354b in ?? () from /lib/libc.so.6
(gdb)
#4 0x00007fb8734c6c7c in free () from /lib/libc.so.6
(gdb)
#5 0x00007fb873b7cb91 in __tcf_3 () from /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
(gdb)
#6 0x00007fb873485a4d in __cxa_finalize () from /lib/libc.so.6
(gdb)
#7 0x00007fb873b5a5d3 in __do_global_dtors_aux () from /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
(gdb)
#8 0x00007fff7bddc8e0 in ?? ()
(gdb)
#9 0x00007fb873ba6d01 in _fini () from /Network/Cluster/home/siddharth/OpenMM3/lib/libOpenMM.so
(gdb)
#10 0x0000000000000000 in ?? ()
(gdb)
Initial frame selected; you cannot go up.
(gdb)

}}}

[Michael Sherman]

Yeah, you're right! That's the same as before.

Still, it is slightly more informative. That makes me think it isn't the cleanup of OpenMM objects (System, Integrator, Context) that's failing. Rather, it appears to be failing in the destructor for some global or static object that was allocated in libOpenMM.so.

Peter thought this might be due to incompatible libraries somewhere. For example, if libOpenMM.so were built with different C++ runtime libraries than the ones you're now using, it could conceivably fail while destructing an object like an std::vector or std::string. If there is any chance of this, you might want to delete everything and rebuild from scratch.

Sherm

[Siddharth Srinivasan]

You read my mind Sherm, I did just that and rebuilt everything from scratch, except CUDA, and we have success! Thank you both for your help, I wish I could explain/get an explanation for why it failed, but it works now so I shall move on.

[Peter Eastman]

> I wish I could explain/get an explanation for why it failed

Gremlins. Clearly.

Peter