CudaDevice is always 0

[Michael Schauperl]

Hi,

I have another problem running the OpenMM. I have two gpus in every machine and OpenMM was not able to decide which gpu is free and always tried to run the job on the first gpu. If a program already runs on the first gpu OpenMM is terminated.
As i understood from the Python API i can force OpenMM to use device 0 or 1 by the following command
platform.setPropertyDefaultValue("CudaDevice",devicenumber)
I can actually write every value in devicenumber. I think OpenMM is ignoring it completely.
Does anyone have a suggestion for this issue?
I have seen that there are a few related topics but i was not able to fix my problem whit the suggestions post in there.

Thanks,

Michael

[Peter Eastman]

The property name is "CudaDeviceIndex", not "CudaDevice". (I really need to make it throw an exception if you specify an unknown property.)

Peter

[Michael Schauperl]

Hi Peter,

Thanks for the quick response. I also tried CudaDeviceIndex but the same problem occured. I can set it to 20 and it is still running on the first gpu.
I also tried to set device with OpenMM command. With the same result. So it seems that it is still ignoring this variable.
Actually i found a way around this problem by setting the occupied gpu invisible but it would be still interesting what the problem of the CudaDeviceIndex could be.

Michael

[Peter Eastman]

Hi Michael,

Do you actually have 21 CUDA devices on that machine? If not, then specifying device 20 will certainly not work. What happens if you specify the index of an actual valid device?

After you create your context, try querying that property to see what device it thinks it's using:

Code: Select all

context.getPlatform().getPropertyValue(context, "CudaDeviceIndex")

Peter

[martin spichty]

Hi, Peter,

I have actually the same problem; I have two GPU cards but whatever I specify with

properties = {'CudaDeviceIndex': 'X'}

with X=0,1

has no impact on which GPU card OpenMM is executed; it's always the first card ('0').

Best,
Martin

[Peter Eastman]

Hi Martin,

Just to be absolutely certain, could you post the code you're using both to set and to check the device index? Also, are you certain that both cards are really accessible (e.g. this isn't a shared server with the GPUs in exclusive mode so each one can only be seen by one user)?

What models of GPU are the two cards?

Peter

[martin spichty]

Hi, Peter,

Thanks for your response!
Please find below the information you asked for.

Best,
Martin


Here is the Python code (it is based on the test case simulateAmber.py):
***
from simtk.openmm.app import *
from simtk.openmm import *
from simtk.unit import *
from sys import stdout

platform = Platform.getPlatformByName('CUDA')
properties = {'CudaDeviceIndex': '0'}
prmtop = AmberPrmtopFile('input.prmtop')
inpcrd = AmberInpcrdFile('input.inpcrd')
system = prmtop.createSystem(nonbondedMethod=PME, nonbondedCutoff=1*nanometer, constraints=HBonds)
integrator = LangevinIntegrator(300*kelvin, 1/picosecond, 0.002*picoseconds)
simulation = Simulation(prmtop.topology, system, integrator)
simulation.minimizeEnergy()
simulation.reporters.append(PDBReporter('output.pdb', 1000))
simulation.reporters.append(StateDataReporter(stdout, 1000, step=True, potentialEnergy=True, temperature=True))
simulation.step(1000000)
***

It does not matter what I use for {'CudaDeviceIndex': 'X'} (with X=0,1).

I am using 2 x GTX570 in their default mode;
nvidia-smi -a -i 0 | grep -v "N/A"
gives:
==============NVSMI LOG==============

Timestamp : Mon Jan 27 09:08:44 2014
Driver Version : 331.13

Attached GPUs : 2
GPU 0000:01:00.0
Product Name : GeForce GTX 570
Persistence Mode : Disabled
Driver Model
GPU UUID : GPU-3e38fae1-dcfc-0c02-6580-23125a90d955
VBIOS Version : 70.10.38.00.03
Inforom Version
GPU Operation Mode
PCI
Bus : 0x01
Device : 0x00
Domain : 0x0000
Device Id : 0x108610DE
Bus Id : 0000:01:00.0
Sub System Id : 0x83871043
GPU Link Info
PCIe Generation
Link Width
Fan Speed : 18 %
Memory Usage
Total : 1279 MB
Used : 86 MB
Free : 1193 MB
Compute Mode : Default
Utilization
Ecc Mode
ECC Errors
Volatile
Single Bit
Double Bit
Aggregate
Single Bit
Double Bit
Retired Pages
Temperature
Gpu : 43 C
Power Readings
Clocks
Applications Clocks
Default Applications Clocks
Max Clocks

[Peter Eastman]

How are you determining which GPU it's using? Are you querying CudaDeviceIndex? Or using nvidia-smi? Or something else? Show exactly what you're doing.

Peter

[Lee-Ping Wang]

Hi Martin,

Looking at your script, you set the properties dictionary, but I think you need to use it when creating the Simulation object. Something like:

simulation = Simulation(topology, system, integrator, platform, properties).

Thanks,

- Lee-Ping

[martin spichty]

Hi, Lee-Ping,

Yes, indeed, the proprties wer4e not added in the Simulation object, how embarrassing!
It's now working fine.

Best,
Martin