Pickling

Pickling

One of the main features of Stackless is its ability to pickle and unpickle tasklets. That means that a running program inside a tasklet can be persistently stored to a file or string. Later, it can be restored again and can continue to run at the point where it was previously halted. This need not be on the same machine!

Example - Pickling and unpickling simple tasklets:

from stackless import run, schedule, tasklet
import pickle

def aCallable(name):
    print "  aCallable<%s>: Before schedule()" % name
    schedule()
    print "  aCallable<%s>: After schedule()" % name

tasks = []
for name in "ABCDE":
    tasks.append(tasklet(aCallable)(name))

print "Schedule 1:"
schedule()

print
print "Pickling..."
pickledTasks = pickle.dumps(tasks)

print
print "Schedule 2:"
schedule()

unpickledTasks = pickle.loads(pickledTasks)
for task in unpickledTasks:
    task.insert()
print
print "Schedule Unpickled Tasks:"
schedule()

Custom Tasklet or Channel Subclasses

Subclassing the tasklet or channel classes is pretty straightforward. In fact, if you need to persist any extra information with your tasklets or channels, you will need to do this. Neither of the classes have their slots or instance dictionary persisted with them when pickled.

The following example should illustrate exactly what the problem is.

Example - Named tasklets:

class CustomTasklet(stackless.tasklet):
    name = "UNNAMED"

    def __str__(self):
        return "<CustomTasklet(name='%s')>" % self.name

def f(): pass

t1 = CustomTasklet(f)()
t1.name = "Pointless"

s = pickle.dumps(t1)
t2 = pickle.loads(s)

Output:

>>> print t2.name == t1.name
False
>>> print t2.name
'UNNAMED'

However, you can easily get around this by overriding __reduce__() / __reduce_ex__() and __setstate__() to add whatever else you want to be persisted.

Example - Named tasklets that persist their name:

class CustomTasklet(stackless.tasklet):
    name = "UNNAMED"

    def __str__(self):
        return "<CustomTasklet(name='%s')>" % self.name

    # When this is present, it is called in lieu of __reduce__.
    # As the base tasklet class provides it, we need to as well.
    def __reduce_ex__(self, pickleVersion):
        return self.__reduce__()

    def __reduce__(self):
        # Get into the list that will eventually be returned to
        # __setstate__ and append our own entry into it (the
        # dictionary of instance variables).
        ret = list(stackless.tasklet.__reduce__(self))
        l = list(ret[2])
        l.append(self.__dict__)
        ret[2] = tuple(l)
        return tuple(ret)

    def __setstate__(self, l):
        # Update the instance dictionary with the value we added in.
        self.__dict__.update(l[-1])
        # Let the tasklet get on with being reconstituted by giving
        # it the original list (removing our addition).
        return stackless.tasklet.__setstate__(self, l[:-1])

def f(): pass

t1 = CustomTasklet(f)()
t1.name = "Pointless"

s = pickle.dumps(t1)
t2 = pickle.loads(s)

Output:

>>> print t2.name == t1.name
True
>>> print t2.name
'Pointless'

Tasklets

Pickling a tasklet that is blocked on a channel, will not result in the pickling of that channel unless you are explicitly pickling a reference to that channel along with it.

If a tasklet is blocked on a channel that is not in any scope contained in the function the tasklet is bound to, then it will not be pickled.

Example - Channel not pickled:

def f():
    c.receive()

c = stackless.channel()
t1 = stackless.tasklet(f)()

s = pickle.dumps(t1)
t2 = pickle.loads(s)

# The tasklet will not be attached to a channel.
assert t2._channel is None

Example - Channel pickled:

def f(c):
    c.receive()

c = stackless.channel()
t1 = stackless.tasklet(f)(c)

s = pickle.dumps(t1)
t2 = pickle.loads(s)

# The tasklet will be attached to a channel.
assert t2._channel is not None

Channels

Pickling a channel, will also pickle any tasklets currently blocked on it. But sometimes you just want to pickle the channel with only some of those tasklets still blocked on it, or perhaps, tasklets you isolate from the channel on their own. You can do this by using __reduce__() and __setstate__().

Example - Removing a tasklet from a channel and pickling the results:

# Given a channel 'c' with four tasklets blocked on it, where
# we want to just pickle the first.

# Get the channel state.
x, y, (balance, flags, tasklets) = c.__reduce__()

# Get the tasklet and remove it from the ones on the channel.
t = tasklets[0]
del tasklets[0]

# Rebuild the channel without the tasklet.  You do not need to
# bother adjusting the balance for the changes you made to the
# list of blocked tasklets, as it is recalculated automatically.
# This will replace the channels existing state.  But if you
# want to keep the channel as it is, you can create a new
# and use it in place of 'c'.
c.__setstate__((balance, flags, tasklets))

# Pickle just the tasklet (and whatever it holds a reference to).
s1 = pickle.dumps(t)
# Pickle the channel which no longer has that channel.
s2 = pickle.dumps(c)

Two important things to know about the values you get from __reduce__() and pass to __setstate__():

• The list of tasklets, featured in both functions, is in the order in which the tasklets came to be blocked on the channel.
• When passed to __setstate__() the actual balance value is not used, except for its sign, which is used to indicate whether the blocked tasklets are receiving or sending.