OpenMM is too slow on Titan X on Sherlock

[Anton Sinitskiy]

I've run a benchmark simulation of Dihydrofolate Reductase using one Titan X GPU (CUDA platform) on sherlock.stanford.edu. It gave me only 109 ns/day, in comparison to 174 ns/day reported here: http://wiki.simtk.org/openmm/BenchmarkOpenMMDHFR

Specifically, I submitted the following job script:

#!/bin/bash
#BATCH --job-name="xxx"
#SBATCH -o ...somewhere_on_sherlock....out
#SBATCH -e ...somewhere_on_sherlock....err
#SBATCH --partition=gpu
#SBATCH --qos=gpu
#SBATCH -n 1
#SBATCH -t 0:05:00
#SBATCH --gres=gpu:gtx:1
module load cuda/7.0
cd ...somewhere_on_sherlock...
echo "CUDA_VISIBLE_DEVICES $CUDA_VISIBLE_DEVICES"
python benchmark.py --platform=CUDA --test=pme --pme-cutoff=0.9
wait

to the gpu queue on sherlock. The output was:

CUDA_VISIBLE_DEVICES 6
Platform: CUDA
Precision: single
Test: pme (cutoff=0.9)
Step Size: 2 fs
Integrated 36452 steps in 57.6882 seconds
109.189 ns/day

I don't have this problem with speed on CPU: when I typed in a command line on a log-in node on Sherlock

python benchmark.py --platform=CPU --test=pme --pme-cutoff=0.8

I got:

Platform: CPU
Test: pme (cutoff=0.8)
Step Size: 2 fs
Integrated 4258 steps in 59.3001 seconds
12.4078 ns/day

which is a bit faster than 10.5 ns/day reported at http://wiki.simtk.org/openmm/BenchmarkOpenMMDHFR.

I've also run simulations of my system with ~66K atoms that I previously ran in Amber on Sherlock on one Titan X GPU. In Amber, I had ~60 ns/day, and in OpenMM I've got only 28 ns/day.

Do you guys have any ideas why is it so slow on GPUs on Sherlock?

[NIKOLAY PLOTNIKOV]

Anton,
did you figure out what was the problem?
Did you use pre-compiled binaries or build them yourself?

[Peter Eastman]

It turned out he wasn't running on a Titan X. Sherlock actually has two different GPU queues, one with Titan X and the other with something else (I'm not certain what, but something slower).

Peter