CudaDevice is always 0
- Michael Schauperl
- Posts: 22
- Joined: Wed Sep 04, 2013 6:10 am
CudaDevice is always 0
Hi,
I have another problem running the OpenMM. I have two gpus in every machine and OpenMM was not able to decide which gpu is free and always tried to run the job on the first gpu. If a program already runs on the first gpu OpenMM is terminated.
As i understood from the Python API i can force OpenMM to use device 0 or 1 by the following command
platform.setPropertyDefaultValue("CudaDevice",devicenumber)
I can actually write every value in devicenumber. I think OpenMM is ignoring it completely.
Does anyone have a suggestion for this issue?
I have seen that there are a few related topics but i was not able to fix my problem whit the suggestions post in there.
Thanks,
Michael
I have another problem running the OpenMM. I have two gpus in every machine and OpenMM was not able to decide which gpu is free and always tried to run the job on the first gpu. If a program already runs on the first gpu OpenMM is terminated.
As i understood from the Python API i can force OpenMM to use device 0 or 1 by the following command
platform.setPropertyDefaultValue("CudaDevice",devicenumber)
I can actually write every value in devicenumber. I think OpenMM is ignoring it completely.
Does anyone have a suggestion for this issue?
I have seen that there are a few related topics but i was not able to fix my problem whit the suggestions post in there.
Thanks,
Michael
- Peter Eastman
- Posts: 2573
- Joined: Thu Aug 09, 2007 1:25 pm
Re: CudaDevice is always 0
The property name is "CudaDeviceIndex", not "CudaDevice". (I really need to make it throw an exception if you specify an unknown property.)
Peter
Peter
- Michael Schauperl
- Posts: 22
- Joined: Wed Sep 04, 2013 6:10 am
Re: CudaDevice is always 0
Hi Peter,
Thanks for the quick response. I also tried CudaDeviceIndex but the same problem occured. I can set it to 20 and it is still running on the first gpu.
I also tried to set device with OpenMM command. With the same result. So it seems that it is still ignoring this variable.
Actually i found a way around this problem by setting the occupied gpu invisible but it would be still interesting what the problem of the CudaDeviceIndex could be.
Michael
Thanks for the quick response. I also tried CudaDeviceIndex but the same problem occured. I can set it to 20 and it is still running on the first gpu.
I also tried to set device with OpenMM command. With the same result. So it seems that it is still ignoring this variable.
Actually i found a way around this problem by setting the occupied gpu invisible but it would be still interesting what the problem of the CudaDeviceIndex could be.
Michael
- Peter Eastman
- Posts: 2573
- Joined: Thu Aug 09, 2007 1:25 pm
Re: CudaDevice is always 0
Hi Michael,
Do you actually have 21 CUDA devices on that machine? If not, then specifying device 20 will certainly not work. What happens if you specify the index of an actual valid device?
After you create your context, try querying that property to see what device it thinks it's using:
Peter
Do you actually have 21 CUDA devices on that machine? If not, then specifying device 20 will certainly not work. What happens if you specify the index of an actual valid device?
After you create your context, try querying that property to see what device it thinks it's using:
Code: Select all
context.getPlatform().getPropertyValue(context, "CudaDeviceIndex")
- martin spichty
- Posts: 10
- Joined: Thu Jul 23, 2009 12:36 am
Re: CudaDevice is always 0
Hi, Peter,
I have actually the same problem; I have two GPU cards but whatever I specify with
properties = {'CudaDeviceIndex': 'X'}
with X=0,1
has no impact on which GPU card OpenMM is executed; it's always the first card ('0').
Best,
Martin
I have actually the same problem; I have two GPU cards but whatever I specify with
properties = {'CudaDeviceIndex': 'X'}
with X=0,1
has no impact on which GPU card OpenMM is executed; it's always the first card ('0').
Best,
Martin
- Peter Eastman
- Posts: 2573
- Joined: Thu Aug 09, 2007 1:25 pm
Re: CudaDevice is always 0
Hi Martin,
Just to be absolutely certain, could you post the code you're using both to set and to check the device index? Also, are you certain that both cards are really accessible (e.g. this isn't a shared server with the GPUs in exclusive mode so each one can only be seen by one user)?
What models of GPU are the two cards?
Peter
Just to be absolutely certain, could you post the code you're using both to set and to check the device index? Also, are you certain that both cards are really accessible (e.g. this isn't a shared server with the GPUs in exclusive mode so each one can only be seen by one user)?
What models of GPU are the two cards?
Peter
- martin spichty
- Posts: 10
- Joined: Thu Jul 23, 2009 12:36 am
Re: CudaDevice is always 0
Hi, Peter,
Thanks for your response!
Please find below the information you asked for.
Best,
Martin
Here is the Python code (it is based on the test case simulateAmber.py):
***
from simtk.openmm.app import *
from simtk.openmm import *
from simtk.unit import *
from sys import stdout
platform = Platform.getPlatformByName('CUDA')
properties = {'CudaDeviceIndex': '0'}
prmtop = AmberPrmtopFile('input.prmtop')
inpcrd = AmberInpcrdFile('input.inpcrd')
system = prmtop.createSystem(nonbondedMethod=PME, nonbondedCutoff=1*nanometer, constraints=HBonds)
integrator = LangevinIntegrator(300*kelvin, 1/picosecond, 0.002*picoseconds)
simulation = Simulation(prmtop.topology, system, integrator)
simulation.minimizeEnergy()
simulation.reporters.append(PDBReporter('output.pdb', 1000))
simulation.reporters.append(StateDataReporter(stdout, 1000, step=True, potentialEnergy=True, temperature=True))
simulation.step(1000000)
***
It does not matter what I use for {'CudaDeviceIndex': 'X'} (with X=0,1).
I am using 2 x GTX570 in their default mode;
nvidia-smi -a -i 0 | grep -v "N/A"
gives:
==============NVSMI LOG==============
Timestamp : Mon Jan 27 09:08:44 2014
Driver Version : 331.13
Attached GPUs : 2
GPU 0000:01:00.0
Product Name : GeForce GTX 570
Persistence Mode : Disabled
Driver Model
GPU UUID : GPU-3e38fae1-dcfc-0c02-6580-23125a90d955
VBIOS Version : 70.10.38.00.03
Inforom Version
GPU Operation Mode
PCI
Bus : 0x01
Device : 0x00
Domain : 0x0000
Device Id : 0x108610DE
Bus Id : 0000:01:00.0
Sub System Id : 0x83871043
GPU Link Info
PCIe Generation
Link Width
Fan Speed : 18 %
Memory Usage
Total : 1279 MB
Used : 86 MB
Free : 1193 MB
Compute Mode : Default
Utilization
Ecc Mode
ECC Errors
Volatile
Single Bit
Double Bit
Aggregate
Single Bit
Double Bit
Retired Pages
Temperature
Gpu : 43 C
Power Readings
Clocks
Applications Clocks
Default Applications Clocks
Max Clocks
Thanks for your response!
Please find below the information you asked for.
Best,
Martin
Here is the Python code (it is based on the test case simulateAmber.py):
***
from simtk.openmm.app import *
from simtk.openmm import *
from simtk.unit import *
from sys import stdout
platform = Platform.getPlatformByName('CUDA')
properties = {'CudaDeviceIndex': '0'}
prmtop = AmberPrmtopFile('input.prmtop')
inpcrd = AmberInpcrdFile('input.inpcrd')
system = prmtop.createSystem(nonbondedMethod=PME, nonbondedCutoff=1*nanometer, constraints=HBonds)
integrator = LangevinIntegrator(300*kelvin, 1/picosecond, 0.002*picoseconds)
simulation = Simulation(prmtop.topology, system, integrator)
simulation.minimizeEnergy()
simulation.reporters.append(PDBReporter('output.pdb', 1000))
simulation.reporters.append(StateDataReporter(stdout, 1000, step=True, potentialEnergy=True, temperature=True))
simulation.step(1000000)
***
It does not matter what I use for {'CudaDeviceIndex': 'X'} (with X=0,1).
I am using 2 x GTX570 in their default mode;
nvidia-smi -a -i 0 | grep -v "N/A"
gives:
==============NVSMI LOG==============
Timestamp : Mon Jan 27 09:08:44 2014
Driver Version : 331.13
Attached GPUs : 2
GPU 0000:01:00.0
Product Name : GeForce GTX 570
Persistence Mode : Disabled
Driver Model
GPU UUID : GPU-3e38fae1-dcfc-0c02-6580-23125a90d955
VBIOS Version : 70.10.38.00.03
Inforom Version
GPU Operation Mode
PCI
Bus : 0x01
Device : 0x00
Domain : 0x0000
Device Id : 0x108610DE
Bus Id : 0000:01:00.0
Sub System Id : 0x83871043
GPU Link Info
PCIe Generation
Link Width
Fan Speed : 18 %
Memory Usage
Total : 1279 MB
Used : 86 MB
Free : 1193 MB
Compute Mode : Default
Utilization
Ecc Mode
ECC Errors
Volatile
Single Bit
Double Bit
Aggregate
Single Bit
Double Bit
Retired Pages
Temperature
Gpu : 43 C
Power Readings
Clocks
Applications Clocks
Default Applications Clocks
Max Clocks
- Peter Eastman
- Posts: 2573
- Joined: Thu Aug 09, 2007 1:25 pm
Re: CudaDevice is always 0
How are you determining which GPU it's using? Are you querying CudaDeviceIndex? Or using nvidia-smi? Or something else? Show exactly what you're doing.
Peter
Peter
- Lee-Ping Wang
- Posts: 102
- Joined: Sun Jun 19, 2011 5:14 pm
Re: CudaDevice is always 0
Hi Martin,
Looking at your script, you set the properties dictionary, but I think you need to use it when creating the Simulation object. Something like:
simulation = Simulation(topology, system, integrator, platform, properties).
Thanks,
- Lee-Ping
Looking at your script, you set the properties dictionary, but I think you need to use it when creating the Simulation object. Something like:
simulation = Simulation(topology, system, integrator, platform, properties).
Thanks,
- Lee-Ping
- martin spichty
- Posts: 10
- Joined: Thu Jul 23, 2009 12:36 am
Re: CudaDevice is always 0
Hi, Lee-Ping,
Yes, indeed, the proprties wer4e not added in the Simulation object, how embarrassing!
It's now working fine.
Best,
Martin
Yes, indeed, the proprties wer4e not added in the Simulation object, how embarrassing!
It's now working fine.
Best,
Martin