global imports

[hyukim17]

I know that dill can have issues pickling global variables, but I was wondering if the following case is intended. My use case for this is setting up a job on a given machine, pickling the instructions, sending it over to remote machines, unpickling the objects, and running the jobs there.

Run the following in one file.

import dill
import numpy as np

def test():
  return np

if __name__ == "__main__":
  with open("out", "w") as f:
    f.write(dill.dumps(test))

Run the next in a different file. Running in a different file is necessary to emulate the environment I am using, where the numpy import is not included at the target machine.

import dill

with open("out") as f:
  contents = f.read()

res = dill.loads(contents)

print(res())

This yields:

NameError: global name 'np' is not defined

However, the code works if, instead of the first file, I define the function test in a separate file (so that it is in a formal namespace, not "main", load and pickle it from a separate runner, the numpy import also gets packaged and unpickled successfully.

In other words, if we have a file

import numpy as np

def test():
  return np

and we have a separate file

from source import test

with open("out", "w") as f:
  f.write(dill.dumps(test))

then the unpickling is fine.

As it turns out, setting dill.settings["recurse"] = True fixes this particular example, but in my actual script, which is more complicated, doing so raises:

RuntimeError: maximum recursion depth exceeded in cmp

Can I get global imports such that:

1. I do not have to define a separate file for the function I am pickling
2. I do not trigger maximum recursion depth?

[mmckerns]

The answer to (1) is: use dill.settings['recurse']=True. This will likely become the default setting, however it currently is not.

I can't answer (2) as posed. Can you give an example that raises the recursion error? That would help see what needs to be done.

[mmckerns]

as far as I can tell, this is a duplicate of #123

[jessanmen1]

Just wanted to add that dill.settings['recurse']=True solved an issue like the described before while dilling scikit-learn pipelines. Thanks.