OpenMM energy conservation test with AMBER + OBC GBSA for T4 lysozyme L99A (parm96) bound to 1-methylpyrrole (GAFF + AM1-BCC charges) - 2616 atoms

John D. Chodera
jchodera@berkeley.edu
QB3-Berkeley Fellow, University of California, Berkeley
email: jchodera@gmail.com
mobile: 415.867.7384
url: http://www.choderalab.org

Note that you need to set the OPENMM_INSTALL_DIR environment variable to the appropriate installation directory (such as the default: /usr/local/openmm) in order for the CMakeLists.txt to work correctly.

COMPILATION AND RUNNING

# generate Makefile
cmake .
# clean up old files
rm -f forcecomparison
# build executable
make forcecomparison
# run (will compare forces between Reference and Cuda platforms if plugins are set up correctly)
./forcecomparison

BUG REPORT

What appeared to initially be poor energy conservation on my MacBook Pro appears to be a widespread platform-dependent issue with incorrect force calculation for large systems.  The attached test code will compare forces computed for T4 lysozyme L99A (AMBER parm96) + 1-methylpyrrole (GAFF + AM1-BCC) in OBC GBSA (2616 atoms) on both Reference and, if available, Cuda platforms, and report the RMS in kJ/mol/nm/atom.

On several systems, we have observed this RMS to be small (<< 1 kJ/mol/nm/atom), but on several other systems, this RMS is very large (> 80 kJ/mol/nm/atom) which leads to rapid heating with the VerletIntegrator, but may not be noticeable with the LangevinIntegrator or when using an AndersenThermostat.

I think it's clear that the problem we're seeing only manifests itself on large systems.  The gradient error on small systems (like a ligand) is tiny on all platforms / driver combinations tested.

I spent the evening cycling through complete uninstall/install/rebuild cycles for CUDA toolkit and SDK versions 2.0, 2.1, and 2.2 on my month-old MacBook Pro.  After each reinstall, I rebuild OpenMM from scratch and then rebuild by gradient test code.  These tests were all performed with OpenMM libraries compiled for source, svn rev 1432, 2009-06-30 10:14:36 -0700 (Tue, 30 Jun 2009).

MacBook Pro + NVIDIA GeForce 9600M GT 128MB + CUDA toolkit 2.0 + SDK 2.0 : RMS 80.701 kJ/mol/nm (*)
MacBook Pro + NVIDIA GeForce 9600M GT 128MB + CUDA toolkit 2.1 + SDK 2.1 : RMS 80.701 kJ/mol/nm (**)
MacBook Pro + NVIDIA GeForce 9600M GT 128MB + CUDA toolkit 2.2 + SDK 2.2 : RMS 80.701 kJ/mol/nm (***)

(*) TestReferenceBrownianIntegrator failed
(**) TestCudaVariableVerletIntegrator and TestCudaVerletIntegrator failed
(***) all CUDA and OpenMM tests pass

However, the Mac Pro I just placed an NVIDIA GeForce GTX 285 in works perfectly:

Mac Pro 2008 + NVIDIA GeForce GTX 285 1GB + CUDA toolkit 2.2.1 + SDK 2.21 : RMS 0.047 kJ/mol/nm (***)

Kim's Mac Pro at Vertex also won't run the energy conservation test properly, so it would presumably fail this gradient test as well (haven't had a chance to check yet):

MacBook Pro + NVIDIA GeForce GT 120 + CUDA toolkit 2.1 + SDK 2.1 : fails energy conservation (***)

Imran Haque's Linux machine:

Intel Linux + NVIDIA GeForce GTS 250 + CUDA 2.2 + display driver 185.18.08-beta + Ubuntu 9.04 : 0.047 kJ/mol/nm (****)

(****) TestCudaNonbonded failed (!!!)

The results are suggestive, to Imran, that the GPU width might play a role in these errors.  My GeForce 9600M doesn't even pass all the tests, while the 9600M GT passes tests but fails on larger systems, as does Kim's GT 120.  Imran's GTS 250 and my GTX 285 perform well.  Perhaps it is worth testing the code on machines will small GPU widths, such as those with 4 SMs or fewer?