ProcessPool in 3.x fails with PicklingError for dynamically created classes

[benjaminpatrickevans]

Hi there, I'm having some issues getting the ProcessPool to work with custom defined new type objects. A minimal example

import numpy as np
import dill
import pathos

print(pathos.__version__) # '0.2.2.1'
print(dill.__version__) # 0.2.8.2

custom_type = type("ABC", (int, ), {})
instance = custom_type(5)
list_of_instances = [instance] * 10

map(np.add, list_of_instances, [1] * len(list_of_instances)) # Works

dill.copy(list_of_instances) # Works

pool = pathos.multiprocessing.ProcessPool(1)
pool.map(np.add, list_of_instances, [1] * len(list_of_instances)) # Fails, _pickle.PicklingError: Can't pickle <class '__main__.ABC'>: it's not found as __main__.ABC

# And trying another Pool for good luck
pool = pathos.pools.ProcessPool(1)
pool.map(np.add, list_of_instances, [1] * len(list_of_instances)) # Also Fails, _pickle.PicklingError: Can't pickle <class '__main__.ABC'>: it's not found as __main__.ABC

This fails with

_pickle.PicklingError: Can't pickle <class 'main.ABC'>: it's not found as main.ABC

Full stack of the error

_pickle.PicklingError: Can't pickle <class 'main.ABC'>: it's not found as main.ABC
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.7/site-packages/pathos/multiprocessing.py", line 137, in map
return _pool.map(star(f), zip(*args)) # chunksize
File "/usr/local/lib/python3.7/site-packages/multiprocess/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
raise self._value
File "/usr/local/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
put(task)
File "/usr/local/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
cls(buf, protocol).dump(obj)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 786, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 633, in save_reduce
save(cls)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 1293, in save_type
StockPickler.save_global(pickler, obj)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 957, in save_global
(obj, module_name, name)) from None
_pickle.PicklingError: Can't pickle <class 'main.ABC'>: it's not found as main.ABC

If instead custom_type is declared as

class custom_type(int):
    pass

A new error arrises with pool.map

File "/usr/local/lib/python3.7/site-packages/multiprocess/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/lib/python3.7/site-packages/multiprocess/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/multiprocess/pool.py", line 110, in worker
task = get()
File "/usr/local/lib/python3.7/site-packages/multiprocess/queues.py", line 357, in get
return _ForkingPickler.loads(res)
File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 316, in loads
return load(file, ignore)
File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 304, in load
obj = pik.load()
File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 465, in find_class
return StockUnpickler.find_class(self, module, name)
AttributeError: Can't get attribute 'custom_type' on <module 'main' (built-in)>

Yet dill.copy still works perfectly fine in both cases

Any idea what could be going on here?
Thanks!

[mmckerns]

I'm not able to reproduce your error (admittedly, with a slightly newer version of pathos).

>>> import numpy as np
>>> import dill
>>> import pathos
>>> 
>>> print(pathos.__version__) # '0.2.2.1'
0.2.3.dev0
>>> print(dill.__version__) # 0.2.8.2
0.2.8.2
>>> 
>>> custom_type = type("ABC", (int, ), {})
>>> instance = custom_type(5)
>>> list_of_instances = [instance] * 10
>>> 
>>> map(np.add, list_of_instances, [1] * len(list_of_instances)) # Works
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>> 
>>> dill.copy(list_of_instances)
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> 
>>> dill.check(list_of_instances)
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> pool = pathos.multiprocessing.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>> pool = pathos.pools.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>> 

[mmckerns]

Oh, wait... I was using 2.7. I see... it does fail for 3.7.

[mmckerns]

>>> dill.detect.trace(True)
>>> dill.check(list_of_instances)
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x10c777048>
# F2
T1: <class 'type'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
T1: <class 'int'>
# T1
D2: <dict object at 0x10c9252d0>
# D2
# T2
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> 

And the failure case...

>>> pool = pathos.pools.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
F2: <function mapstar at 0x10c7eb620>
# F2
F1: <function starargs.<locals>.<lambda> at 0x10c91d400>
F2: <function _create_function at 0x10c7770d0>
# F2
Co: <code object <lambda> at 0x10c7e8d20, file "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.3.dev0-py3.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <class 'code'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
# Co
D4: <dict object at 0x10c7ec5a0>
# D4
Ce: <cell at 0x10c8eed98: numpy.ufunc object at 0x7fd98bd15700>
F2: <function _create_cell at 0x10c777488>
# F2
F2: <function _ufunc_reconstruct at 0x10be39d08>
# F2
# Ce
D2: <dict object at 0x10c938510>
# D2
# F1
T5: <class '__main__.ABC'>
F2: <function mapstar at 0x10c7eb620>
# F2
F1: <function starargs.<locals>.<lambda> at 0x10c91d400>
F2: <function _create_function at 0x10c7770d0>
# F2
Co: <code object <lambda> at 0x10c7e8d20, file "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.3.dev0-py3.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <class 'code'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
# Co
D4: <dict object at 0x10c7ec5a0>
# D4
Ce: <cell at 0x10c8eed98: numpy.ufunc object at 0x7fd98bd15700>
F2: <function _create_cell at 0x10c777488>
# F2
F2: <function _ufunc_reconstruct at 0x10be39d08>
# F2
# Ce
D2: <dict object at 0x10c938510>
# D2
# F1
T5: <class '__main__.ABC'>
F2: <function mapstar at 0x10c7eb620>
# F2
F1: <function starargs.<locals>.<lambda> at 0x10c91d400>
F2: <function _create_function at 0x10c7770d0>
# F2
Co: <code object <lambda> at 0x10c7e8d20, file "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.3.dev0-py3.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <class 'code'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
# Co
D4: <dict object at 0x10c7ec5a0>
# D4
Ce: <cell at 0x10c8eed98: numpy.ufunc object at 0x7fd98bd15700>
F2: <function _create_cell at 0x10c777488>
# F2
F2: <function _ufunc_reconstruct at 0x10be39d08>
# F2
# Ce
D2: <dict object at 0x10c938510>
# D2
# F1
T5: <class '__main__.ABC'>
F2: <function mapstar at 0x10c7eb620>
# F2
F1: <function starargs.<locals>.<lambda> at 0x10c91d400>
F2: <function _create_function at 0x10c7770d0>
# F2
Co: <code object <lambda> at 0x10c7e8d20, file "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.3.dev0-py3.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <class 'code'>
F2: <function _load_type at 0x10c76ff28>
# F2
# T1
# Co
D4: <dict object at 0x10c7ec5a0>
# D4
Ce: <cell at 0x10c8eed98: numpy.ufunc object at 0x7fd98bd15700>
F2: <function _create_cell at 0x10c777488>
# F2
F2: <function _ufunc_reconstruct at 0x10be39d08>
# F2
# Ce
D2: <dict object at 0x10c938510>
# D2
# F1
T5: <class '__main__.ABC'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
 ...
_pickle.PicklingError: Can't pickle <class '__main__.ABC'>: it's not found as __main__.ABC

[mmckerns]

In 2.7:

>>> dill.detect.trace(True)
>>> dill.check(list_of_instances)
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
F2: <function _load_type at 0x105cb0938>
# F2
# T1
T1: <type 'int'>
# T1
D2: <dict object at 0x105da1e88>
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> 

and...

>>> pool = pathos.pools.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
F2: <function mapstar at 0x105d3b410>
# F2
F1: <function <lambda> at 0x105dc9aa0>
F2: <function _create_function at 0x105cb0a28>
# F2
Co: <code object <lambda> at 0x105d16bb0, file "/Users/mmckerns/lib/python2.7/site-packages/pathos-0.2.3.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <type 'code'>
F2: <function _load_type at 0x105cb0938>
# F2
# T1
# Co
D4: <dict object at 0x105d64e88>
# D4
Ce: <cell at 0x105dc5210: numpy.ufunc object at 0x7f7f5527be20>
F2: <function _create_cell at 0x105cb0d70>
# F2
F2: <function _ufunc_reconstruct at 0x105768938>
# F2
# Ce
D2: <dict object at 0x105dd0910>
# D2
# F1
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
# T1
T1: <type 'int'>
# T1
D2: <dict object at 0x105dd0e88>
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
D2: <dict object at 0x105dd0d70>
# D2
F2: <function mapstar at 0x105d3b410>
# F2
F1: <function <lambda> at 0x105dc9aa0>
F2: <function _create_function at 0x105cb0a28>
# F2
Co: <code object <lambda> at 0x105d16bb0, file "/Users/mmckerns/lib/python2.7/site-packages/pathos-0.2.3.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <type 'code'>
F2: <function _load_type at 0x105cb0938>
# F2
# T1
# Co
D4: <dict object at 0x105d64e88>
# D4
Ce: <cell at 0x105dc5210: numpy.ufunc object at 0x7f7f5527be20>
F2: <function _create_cell at 0x105cb0d70>
# F2
F2: <function _ufunc_reconstruct at 0x105768938>
# F2
# Ce
D2: <dict object at 0x105dd0910>
# D2
# F1
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
# T1
B2: <built-in function scalar>
T1: <type 'int'>
# T1
# B2
D2: <dict object at 0x105dd07f8>
T4: <type 'numpy.dtype'>
# T4
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
D2: <dict object at 0x105dd06e0>
# D2
F2: <function mapstar at 0x105d3b410>
# F2
F1: <function <lambda> at 0x105dc9aa0>
F2: <function _create_function at 0x105cb0a28>
# F2
Co: <code object <lambda> at 0x105d16bb0, file "/Users/mmckerns/lib/python2.7/site-packages/pathos-0.2.3.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 15>
B2: <built-in function scalar>
T1: <type 'code'>
# B2
F2: <function _load_type at 0x105cb0938>
# F2
T4: <type 'numpy.dtype'>
# T1
# T4
# Co
D4: <dict object at 0x105d64e88>
# D4
Ce: <cell at 0x105dc5210: numpy.ufunc object at 0x7f7f5527be20>
F2: <function _create_cell at 0x105cb0d70>
# F2
F2: <function _ufunc_reconstruct at 0x105768938>
# F2
# Ce
D2: <dict object at 0x105dd0910>
# D2
# F1
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
# T1
T1: <type 'int'>
# T1
D2: <dict object at 0x105dd0c58>
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
D2: <dict object at 0x105dd0d70>
# D2
F2: <function mapstar at 0x105d3b410>
# F2
F1: <function <lambda> at 0x105dc9aa0>
F2: <function _create_function at 0x105cb0a28>
# F2
Co: <code object <lambda> at 0x105d16bb0, file "/Users/mmckerns/lib/python2.7/site-packages/pathos-0.2.3.dev0-py2.7.egg/pathos/helpers/mp_helper.py", line 15>
T1: <type 'code'>
F2: <function _load_type at 0x105cb0938>
B2: <built-in function scalar>
# F2
# B2
# T1
T4: <type 'numpy.dtype'>
# T4
# Co
D4: <dict object at 0x105d64e88>
# D4
Ce: <cell at 0x105dc5210: numpy.ufunc object at 0x7f7f5527be20>
F2: <function _create_cell at 0x105cb0d70>
# F2
F2: <function _ufunc_reconstruct at 0x105768938>
# F2
# Ce
D2: <dict object at 0x105dd0910>
# D2
# F1
T2: <class '__main__.ABC'>
F2: <function _create_type at 0x105cb09b0>
# F2
T1: <type 'type'>
# T1
T1: <type 'int'>
# T1
D2: <dict object at 0x105dd0e88>
# D2
# T2
D2: <dict object at 0x105da1398>
# D2
D2: <dict object at 0x105dd06e0>
# D2
B2: <built-in function scalar>
# B2
T4: <type 'numpy.dtype'>
# T4
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>> 

[mmckerns]

Hmm... the primary difference seems to be that for 3.7, we have T5: <class '__main__.ABC'> which means that the dynamically created class ABC is being passed to pickle (and failing) -- while in 2.7, we have T2: <class '__main__.ABC'>, which uses the dill function _dict_from_dictproxy. Notice that directly using dill also utilizes T2 in both 3.7 and 2.7.

[mmckerns]

Ok, so there's a workaround for now, since this is falling back to pickle, and I have to figure out why it is for 3.7. Python's pickler requires that the name of the dynamically created class is the same as the name of the instance:

>>> import numpy as np
>>> import dill
>>> import pathos
>>> 
>>> print(pathos.__version__) # '0.2.2.1'
0.2.3.dev0
>>> print(dill.__version__) # 0.2.8.2
0.2.9.dev0
>>> 
>>> custom_type = type("custom_type", (int, ), {})
>>> instance = custom_type(5)
>>> list_of_instances = [instance] * 10
>>> 
>>> map(np.add, list_of_instances, [1] * len(list_of_instances)) # Works
<map object at 0x10fd9ab00>
>>> 
>>> dill.check(list_of_instances) # Works
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> 
>>> pool = pathos.pools.ProcessPool(1)
>>> pool.map(np.add, list_of_instances, [1] * len(list_of_instances))
[6, 6, 6, 6, 6, 6, 6, 6, 6, 6]
>>> 

[mmckerns]

This is fixed by uqfoundation/dill@3b043d6. Using the most recent version of dill, your original test code now works for 3.7, etc.

Please reopen this ticket if you are still experiencing an error.

[rakhmaevao]

Hello. I have same error for:

Python: 3.9.7
Pathos: 0.2.8
dill: 0.3.4

Example code:

from abc import abstractmethod, ABC
from multiprocessing import Pool
from pathos.pools import ProcessPool
import pathos
import dill
import platform

print(f'Python: {platform.python_version()}')
print(f'Pathos: {pathos.__version__}')
print(f'dill: {dill.__version__}')

class SomeAbstractClass(ABC):
    @abstractmethod
    def some_methode(self):
        pass


class SomeConreteClass(SomeAbstractClass):
    def __init__(self, blank=None):
        self.some_methode()

    def some_methode(self):
        print('something')


if __name__ == '__main__':
    with ProcessPool(5) as p:
        p.map(SomeConreteClass, [1, 2, 3])

Error output:

Python: 3.9.7
Pathos: 0.2.8
dill: 0.3.4
Traceback (most recent call last):
  File "C:\Users\rahma\Documents\detectionzone\выпаа.py", line 28, in <module>
    p.map(SomeConreteClass, [1, 2, 3])
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\pathos\multiprocessing.py", line 139, in map
    return _pool.map(star(f), zip(*args)) # chunksize
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\pool.py", line 771, in get
    raise self._value
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\pool.py", line 537, in _handle_tasks
    put(task)
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\connection.py", line 214, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\multiprocess\reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 498, in dump
    StockPickler.dump(self, obj)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 487, in dump
    self.save(obj)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 1493, in save_function
    pickler.save_reduce(_create_function, (obj.__code__,
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 692, in save_reduce
    save(args)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 1227, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 692, in save_reduce
    save(args)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 1422, in save_type
    pickler.save_reduce(_create_type, (type(obj), obj.__name__,
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 692, in save_reduce
    save(args)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 886, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 1422, in save_type
    pickler.save_reduce(_create_type, (type(obj), obj.__name__,
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 692, in save_reduce
    save(args)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple
    save(element)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\rahma\Documents\detectionzone\venv\lib\site-packages\dill\_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 997, in _batch_setitems
    save(v)
  File "C:\Users\rahma\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle '_abc._abc_data' object

Process finished with exit code 1

[mmckerns]

@rahmaevao: your issue is not the same as the original issue. Yours is a duplicate of uqfoundation/dill#332.

[Larleyt]

The issue still persists for

Python==3.9
dill==0.3.4
pathos==0.2.8 
mlflow==1.24.0

Basically, in the __init__() of a class I'm trying to pickle there's an mlflow.pyfunc.load_model() function executing. I have a suspicion dill cannot pickle a loaded model to split it across processes correctly. Am I right?
Strangely, build-in multiprocessing package works ok with this case.

Traceback

Traceback (most recent call last): File "project_name/feature_pipeline/main.py", line 7, in application.run() File "project_name/feature_pipeline/application.py", line 55, in run dataframe = feature_calculator.run() File "project_name/feature_pipeline/call_methods.py", line 271, in run self._call_method(method, field_candidates) File "project_name/feature_pipeline/call_methods.py", line 238, in _call_method for dataframe in pool.imap( File "project_name/.venv/lib/python3.9/site-packages/multiprocess/pool.py", line 870, in next raise value File "project_name/.venv/lib/python3.9/site-packages/multiprocess/pool.py", line 537, in _handle_tasks put(task) File "project_name/.venv/lib/python3.9/site-packages/multiprocess/connection.py", line 214, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "project_name/core-tss.ds/.venv/lib/python3.9/site-packages/multiprocess/reduction.py", line 54, in dumps cls(buf, protocol, *args, **kwds).dump(obj) File "project_name/core-tss.ds/.venv/lib/python3.9/site-packages/dill/_dill.py", line 498, in dump StockPickler.dump(self, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 487, in dump self.save(obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 901, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 901, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce save(state) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 1002, in _batch_setitems save(v) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce save(state) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems save(v) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 717, in save_reduce save(state) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems save(v) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 687, in save_reduce save(cls) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 1422, in save_type pickler.save_reduce(_create_type, (type(obj), obj.__name__, File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce save(args) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 901, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 997, in _batch_setitems save(v) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 1493, in save_function pickler.save_reduce(_create_function, (obj.__code__, File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce save(args) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 901, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 1227, in save_cell pickler.save_reduce(_create_cell, (f,), obj=obj) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 692, in save_reduce save(args) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "project_name/.venv/lib/python3.9/site-packages/dill/_dill.py", line 1410, in save_type StockPickler.save_global(pickler, obj, name=name) File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 1070, in save_global raise PicklingError( _pickle.PicklingError: Can't pickle : it's not found as __main__.ClusteringSystem

[mmckerns]

I have a suspicion dill cannot pickle a loaded model to split it across processes correctly. Am I right?

No. dill was build specifically to work across processes (see multiprocess and pathos).

It's a bit hard to determine what's going on with your code when you only posted the traceback (and not the code itself). However, it looks like somewhere down the traceback you are pickling a function, and it's then trying to pickle the global dict, and then there's a thing called ClusteringSystem which it expects to be defined in __main__ but it can't be found. You might try to use dill.settings['recurse'] = True.

However, it does not seem that your issue is the same as the original one, based on the above.