Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix module locals has no builtins #212

Merged
merged 4 commits into from
Oct 10, 2018

Conversation

tomMoral
Copy link
Contributor

Fixes #211

This PR fix the way we detect that a module is importable for local functions.
Note that this not completly fixes the problem in #211 as the issue still appears for dynamic modules eg if you change tests/cloudpickle_test.py from line 419 to

    def test_dynamic_module(self):
        mod = types.ModuleType('mod')
        code = '''
        x = 1
        def f(y):
            range(10)
            return x + y

        class Foo:
            def method(self, x):
                return f(x)
        '''
        exec(textwrap.dedent(code), mod.__dict__)
        mod2 = pickle_depickle(mod, protocol=self.protocol)
        self.assertEqual(mod.x, mod2.x)
        self.assertEqual(mod.f(5), mod2.f(5))
        self.assertEqual(mod.Foo().method(5), mod2.Foo().method(5))

        if platform.python_implementation() != 'PyPy':
            # XXX: this fails with excessive recursion on PyPy.
            mod3 = subprocess_pickle_echo(mod, protocol=self.protocol)
            self.assertEqual(mod.x, mod3.x)
            self.assertEqual(mod.f(5), mod3.f(5))
            self.assertEqual(mod.Foo().method(5), mod3.Foo().method(5))

        # Test dynamic modules when imported back are singletons
        mod1, mod2 = pickle_depickle([mod, mod])
        self.assertEqual(id(mod1), id(mod2))

then the output of the test is

_______________________________________________________________________________________ CloudPickleTest.test_dynamic_module _______________________________________________________________________________________

self = <tests.cloudpickle_test.CloudPickleTest testMethod=test_dynamic_module>

    def test_dynamic_module(self):
        mod = types.ModuleType('mod')
        code = '''
            x = 1
            def f(y):
                range(10)
                return x + y
    
            class Foo:
                def method(self, x):
                    return f(x)
            '''
        exec(textwrap.dedent(code), mod.__dict__)
        mod2 = pickle_depickle(mod, protocol=self.protocol)
        self.assertEqual(mod.x, mod2.x)
        self.assertEqual(mod.f(5), mod2.f(5))
        self.assertEqual(mod.Foo().method(5), mod2.Foo().method(5))
    
        if platform.python_implementation() != 'PyPy':
            # XXX: this fails with excessive recursion on PyPy.
            mod3 = subprocess_pickle_echo(mod, protocol=self.protocol)
            self.assertEqual(mod.x, mod3.x)
            self.assertEqual(mod.f(5), mod3.f(5))
            self.assertEqual(mod.Foo().method(5), mod3.Foo().method(5))
    
        # Test dynamic modules when imported back are singletons
>       mod1, mod2 = pickle_depickle([mod, mod])

code       = '\n        x = 1\n        def f(y):\n            range(10)\n            return x + y\n\n        class Foo:\n            def method(self, x):\n                return f(x)\n        '
mod        = <module 'mod'>
mod2       = <module 'mod'>
mod3       = <module 'mod'>
self       = <tests.cloudpickle_test.CloudPickleTest testMethod=test_dynamic_module>

tests/cloudpickle_test.py:445: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/cloudpickle_test.py:69: in pickle_depickle
    return pickle.loads(cloudpickle.dumps(obj, protocol=protocol))
cloudpickle/cloudpickle.py:931: in dumps
    cp.dump(obj)
cloudpickle/cloudpickle.py:284: in dump
    return Pickler.dump(self, obj)
/usr/lib/python3.4/pickle.py:412: in dump
    self.save(obj)
/usr/lib/python3.4/pickle.py:479: in save
    f(self, obj) # Call unbound method with explicit self
/usr/lib/python3.4/pickle.py:774: in save_list
    self._batch_appends(obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <cloudpickle.cloudpickle.CloudPickler object at 0x7f63bc9d3550>, items = [<module 'mod'>, <module 'mod'>]

    def _batch_appends(self, items):
        # Helper to batch up APPENDS sequences
        save = self.save
        write = self.write
    
        if not self.bin:
            for x in items:
                save(x)
                write(APPEND)
            return
    
        it = iter(items)
        while True:
>           tmp = list(islice(it, self._BATCHSIZE))
E           KeyError: 'range'

it         = <list_iterator object at 0x7f63bc9d3c50>
items      = [<module 'mod'>, <module 'mod'>]
save       = <bound method CloudPickler.save of <cloudpickle.cloudpickle.CloudPickler object at 0x7f63bc9d3550>>
self       = <cloudpickle.cloudpickle.CloudPickler object at 0x7f63bc9d3550>
write      = <bound method _Framer.write of <pickle._Framer object at 0x7f63bc9d3e10>>

/usr/lib/python3.4/pickle.py:793: KeyError

But I don't think it is possible to fix that as it probably has to do with internal dealing of python for creating a FunctionType object .

@codecov-io
Copy link

codecov-io commented Oct 10, 2018

Codecov Report

Merging #212 into master will increase coverage by 0.05%.
The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #212      +/-   ##
========================================
+ Coverage   84.94%    85%   +0.05%     
========================================
  Files           1      1              
  Lines         578    580       +2     
  Branches      115    114       -1     
========================================
+ Hits          491    493       +2     
  Misses         63     63              
  Partials       24     24
Impacted Files Coverage Δ
cloudpickle/cloudpickle.py 85% <100%> (+0.05%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 355abdb...35ee0d4. Read the comment docs.

@tomMoral tomMoral force-pushed the FIX_module_locals_no_builtins branch from 6d90dd9 to 3656af9 Compare October 10, 2018 14:03
@tomMoral tomMoral force-pushed the FIX_module_locals_no_builtins branch 2 times, most recently from ead8051 to 35ee0d4 Compare October 10, 2018 14:51
Copy link
Contributor

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks very much for the fix.

@ogrisel ogrisel merged commit 51be0f9 into cloudpipe:master Oct 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cloudpickle 0.6.0 is broken on python3.4 for local functions
3 participants