TypeError: can't pickle _abc_data objects #332
Comments
|
@mmckerns here is an minimal, reproducible example: import dill as pickle
from abc import ABCMeta
class TestClass(metaclass=ABCMeta):
pass
with open('test.pickle', 'wb') as f:
pickle.dump(TestClass, f) |
|
And more real world example... from sklearn.compose import ColumnTransformer
class DataFrameTransformer(ColumnTransformer):
def get_feature_names(self):
"""
enable get_feature_names when using remainder passthrough
"""
feature_names = []
for name, transformer, columns in self.transformers_:
if hasattr(transformer, "get_feature_names"):
feature_names += [f"{name}_{x}" for x in transformer.get_feature_names()]
elif transformer == "drop":
continue
elif name == "remainder" and transformer == "passthrough":
feature_names += self._df_columns.take(columns).tolist()
return feature_names
with open('test.pickle', 'wb') as f:
pickle.dump(DataFrameTransformer, f) |
|
I think this is a blocking issue. Cloudpickle handles it fine. I'm not able to migrate production code using dill to the newer python versions since I'm using abstract classes, hence I will need to consider switching to cloudpickle. |
|
I still have the exact same problem. Python 3.8.5, multiprocess 0.70.11.1, dill 0.3.3. Update: The above minimal example runs through with Python 3.6 but throws the above error in Python 3.7, 3.8 and 3.9. |
|
+1 and it's even not easy to debug which object is the cause of the problem |
|
silly workaround (but worked for me): remove all |
|
Any thoughts on how to fix this... or where to start? |
|
I did not realize this was still an issue... it's basically, the |
|
Would it be something like registering those methods you reference similar to the save_XXX methods in dill/_dill.py? |
|
+1 long running convo it seems. |
|
I will take a crack at this this week. |
|
@mmckerns I was able to hack in a solution that solves my broken tests... Anyway, not sure I have this in the right place... but it does fix my test cases. Further input would be welcome! |
|
I think what you have looks reasonable, but should probably be tested a bit more extensively. I think if it's possible to |
|
Yup - @mmckerns it just works with @register(abc.ABCMeta)
def save_abc(pickler, obj):
StockPickler.save_type(pickler, obj)Amazing stuff - I even added a lambda as an instance variable to the test class and dill/pickle still handles both properly! |
|
Feel free to create your own test file for abc, especially if there's a lot to test. |
|
Also having this issue; thanks @emfdavid for taking the time to create a fix. |
|
Any progress on this? |
Hello, the code above doesn't work with the latest. Did I missed something ? |
|
Python 3.6 implements the ABC mechanism in C as opposed to pure Python, so it cannot be pickled the same way as other classes. #450 fixes it, but it wasn't included in 0.3.5 because the feature was not 100% complete by the release date. |
|
If the class inheriting from So one workaround until the inclusion of #450 is to put anything inheriting from pickling.py (the script to execute): import dill
import pickle
class A:
pass
pickle.loads(pickle.dumps(A())) # Succeeds
dill.loads(dill.dumps(A())) # Succeeds
from abc import ABC
class Virtual(ABC):
pass
pickle.loads(pickle.dumps(Virtual())) # Succeeds
dill.loads(dill.dumps(Virtual())) # Fails
from a_module import VirtualInAnotherFile
pickle.loads(pickle.dumps(VirtualInAnotherFile())) # Succeeds
dill.loads(dill.dumps(VirtualInAnotherFile())) # Succeedsa_module.py: from abc import ABC
class VirtualInAnotherFile(ABC):
passWith at which point you get a The trace for the |
|
Without updating the dill codebase, it is not possible to be able to pickle any ABC class completely. This short segment of code, however, can cover most cases: import dill, _abc
@dill.register(_abc._abc_data)
def save_abc_impl(pickler, obj):
pickler.save(None)Unfortunately, this workaround throws away all metadata Python keeps about the ABC class, including registered subclasses, so don't use this work around if the ABCs you are pickling use this feature. Doing so, will lead to bizarre behavior. >>> import abc
>>> import dill
>>>
>>> class A(abc.ABC):
... @abc.abstractmethod
... def f(self):
... pass
...
>>> class B:
... def f():
... print('h')
...
>>> A.register(B)
>>>
>>> issubclass(B, dill.copy(A))
False
>>> issubclass(B, A)
True |
aren't pickleable due to: uqfoundation/dill#332 Strangely, I can't reproduce this within a pytest test. It came about when I defined a class deriving from Object within a jupyter environment, so it may have something to do with the `__main__` namespace. I wasn't able to replicate this though within a pytest.
|
I haven't experienced this issue before, however now with the release of Dill 0.3.6, I also get this error. I can't quite pin point it to a reproducible snippet yet, but my code base has been using abstract base classes which are also part of the serialized objects in various places without issues until now. Downgrading to Dill 0.3.5.1 fixes it for me. |
|
@NiklasRosenstein: It's my understanding (as demonstrated in this and other issues) that |
|
as @NiklasRosenstein writes, this worked for me as well.
@mmckerns it is interesting that downgrading dill alone fixes the issue, hinting that something in dill 0.3.6 changed that surfaces this issue |
|
@miraculixx: I agree that some recent change may have brought this issue to the surface. My point is that the |
|
It appears like it was left dangling, thanks. |
|
I still have this issue with Python 3.10 |
|
I'm not seeing an issue with 3.10. Can you provide an example that demonstrates the issue? |
|
Still facing this issue with Python 3.8 and dill 3.6 The following example still returns "TypeError: cannot pickle '_abc_data' object" |
|
Here's your example using If you update to master do you still see the issue? |
|
any updates on a fix? 👉👈🥺 |
On Python 3.7, I'm getting this error:
I believe it may be related to this issue: cloudpipe/cloudpickle#180
The text was updated successfully, but these errors were encountered: