Share 
Follow 
AboutDownloadsForumsSource CodeIssues
Date:
2010-08-06 21:20
Priority:
3
State:
Open
Submitted by:
John Chodera (jchodera)
Assigned to:
Christopher Bruns (cmbruns)
Summary:
Error when unpickling Quantity objects

Detailed description
It appears that Quantity objects can be pickled without a problem, unpickling causes an error that the recursion limit has been exceeded. The demo code demonstrates this issue on my system (Python 2.6.4, numpy 1.4.0), though the error message thrown is not very illuminating.

I originally discovered this problem in my efforts to use Parallel Python to parallelize some pyopenmm code. (Note that I'm not transporting any Swig objects over the network that shouldn't be pickleable.) When my (complex) Parallel Python code fails, I get a aomewhat more illuminating error message suggesting a problem with the delegation of __getattr__ by Quantity:

File "/Users/jchodera/local/python/lib/python2.6/site-packages/simtk/unit/quantity.py",
line 163, in __getattr__
ret_val = getattr(self._value, attribute)

Simple demo code follows.

----

import simtk.unit
import numpy

x = numpy.zeros([10], numpy.float64)
quantity = simtk.unit.Quantity(x, simtk.unit.amu)

import pickle

outfile = open('pickle_test.pkl', 'wb')
pickle.dump(quantity, outfile)
outfile.close()

infile = open('pickle_test.pkl', 'rb')
new_quantity = pickle.load(infile)
infile.close()

Add A Comment: Notepad

Message  ↓
Date: 2010-08-08 13:13
Sender: John Chodera

My temporary fix seemed to be working without ill effects, so I've checked it in as r780. As previously mentioned, this is worth a second look to make sure I have pickled all necessary information. Efficiency gains could also be had by re-examining what information is stored when the unit field is pickled; right now, it seems like pickling even a simple Quantity(0.0, units.amu) takes up 32K!

Date: 2010-08-07 05:59
Sender: John Chodera

The addition of __setstate__ and __getstate__ code below seems to have fixed the simple example (also revised, below). The resulting picked file is still large, suggesting further optimizations could be had by also adding __setstate__ and __getstate__ methods to the simtk.unit.Unit class to be more parsimonious in the information stored, but this is probably beyond my ken.

I'll test this more thoroughly before checking in a change. Thanks for the advice!

John

--

def __getstate__(self):
state = dict()
state['_value'] = self._value
state['unit'] = self.unit
return state

def __setstate__(self, state):
self._value = state['_value']
self.unit = state['unit']
return

--
# Simple test of pickling/unpickling

import simtk.unit
import numpy
import sys

x = numpy.zeros([10], numpy.float64)
quantity = simtk.unit.Quantity(x, simtk.unit.kilocalories_per_mole)

import pickle

print "pickling..."
outfile = open('pickle_test.pkl', 'wb')
pickle.dump(quantity, outfile)
outfile.close()
print str(quantity)

print "unpickling..."
infile = open('pickle_test.pkl', 'rb')
new_quantity = pickle.load(infile)
infile.close()

print str(new_quantity)

Date: 2010-08-06 22:00
Sender: Christopher Bruns

That __getattr__ method of quantity is trying to cleverly delegate any unrecognized attribute request from Quantity to the underlying value object. I'm not sure what the trouble is, but it might be illuminating to identify what attribute is being requested here. Then the fix might be as easy as defining that particular attribute for the Quantity class.

Another approach might be to learn whether there are special methods that might be needed to help make a particular class "pickleable".

Unfortunately I might not be able to spend much time on this at the moment.

No Changes Have Been Made to This Item

Feedback