-
Notifications
You must be signed in to change notification settings - Fork 231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wrapt functions are not dill-able #34
Comments
This would probably be something to work out with the So a good place to start is to work with the I've found that |
Thanks so much for your input @mmckerns. So to get started I've set import wrapt
@wrapt.decorator
def pass_through(wrapped, instance, args, kwargs):
return wrapped(*args, **kwargs)
@pass_through
def function():
pass
import dill
dill.detect.trace(True)
f_str = dill.dumps(function) T1: <type 'function'>
INFO:dill:T1: <type 'function'>
F2: <function _load_type at 0x7f4d20b94b90>
INFO:dill:F2: <function _load_type at 0x7f4d20b94b90>
D2: <dict object at 0x7f4d3041cb40>
INFO:dill:D2: <dict object at 0x7f4d3041cb40> assert function == dill.detect.badobjects(function) # True for comparison: def basic_pass_through(f):
def g(*args, **kwargs):
return f(*args, **kwargs)
return g
@basic_pass_through
def basic_function(a):
return 2 * a
import dill
dill.detect.trace(True)
f_str = dill.dumps(basic_function) F1: <function g at 0x7f4d20b226e0>
INFO:dill:F1: <function g at 0x7f4d20b226e0>
F2: <function _create_function at 0x7f4d20b94c80>
INFO:dill:F2: <function _create_function at 0x7f4d20b94c80>
Co: <code object g at 0x7f4d20b337b0, file "<ipython-input-15-3e3fa08cad52>", line 3>
INFO:dill:Co: <code object g at 0x7f4d20b337b0, file "<ipython-input-15-3e3fa08cad52>", line 3>
F2: <function _unmarshal at 0x7f4d20b94b18>
INFO:dill:F2: <function _unmarshal at 0x7f4d20b94b18>
D1: <dict object at 0x7f4d334f8b40>
INFO:dill:D1: <dict object at 0x7f4d334f8b40>
Ce: <cell at 0x7f4d20b323d0: function object at 0x7f4d20b22b18>
INFO:dill:Ce: <cell at 0x7f4d20b323d0: function object at 0x7f4d20b22b18>
F2: <function _create_cell at 0x7f4d20b94f50>
INFO:dill:F2: <function _create_cell at 0x7f4d20b94f50>
F1: <function basic_function at 0x7f4d20b22b18>
INFO:dill:F1: <function basic_function at 0x7f4d20b22b18>
Co: <code object basic_function at 0x7f4d20b332b0, file "<ipython-input-15-3e3fa08cad52>", line 8>
INFO:dill:Co: <code object basic_function at 0x7f4d20b332b0, file "<ipython-input-15-3e3fa08cad52>", line 8>
D1: <dict object at 0x7f4d334f8b40>
INFO:dill:D1: <dict object at 0x7f4d334f8b40>
D2: <dict object at 0x7f4d20b38050>
INFO:dill:D2: <dict object at 0x7f4d20b38050>
D2: <dict object at 0x7f4d20b276e0>
INFO:dill:D2: <dict object at 0x7f4d20b276e0> The first thing of note is that in the basic case dill notices that @mmckerns does this analysis seem correct? If so it seems that the answer to this problem would be to add pickle support to wrapt's special function-like types that it uses, probably be implementing |
@jabooth: I agree with your analysis. I see you options are to either add pickle support (which should be more robust), or to diagnose why the |
For some background reading, you may want to read: The wrapt package doesn't use function closures to implement decorators as that approach is broken in various ways and can't be used to build a really robust decorator. The wrappers are therefore instances of a class, where the class is actually a transparent object proxy with additional function like behaviour built on top to ensure the correct binding behaviour for functions when used in conjunction with methods of a class. There are actually two implementations within wrapt of the object proxy. A C version which is the default and a pure Python fallback version. Right now your tests are probably operating on the C version and why many evil things are happening. That said, even if you installed wrapt without the C extension and allowed it to use the Python version, I am not sure how much further you will get, as the object proxy implementation is quite nasty and overrides many many magic Python methods, as well as doing special setup using meta classes and in the constructor. Not sure that will map well to pickling. |
@jabooth: While |
If you want to experiment to see if the pure Python version of wrapt can be made to work, do:
|
Awesome package, thanks! Implementation and blog are really nice! But the dill / serialization problem is quite nasty especially if one needs to scale a computation with multiprocessing_on_dill or concurrent.futures (e.g. to work on clusters). From a user standpoint, I fallback to decorator.decorator (which I find less nice) but allows for these large computations. For sure not a need of your typical python user, but any library that uses wrapt is not viable for these reasons. |
Tested (python3.6, OSX):
(in a new empty venv): pip install wrapt
(in a new empty venv): WRAPT_EXTENSIONS=false pip install wrapt
If I use decorator.decorator:
Output:
|
Hi! I want to know if there was any update related with this issue since the last comment. I think it would be really interesting to fix this "bug". |
Nothing has changed. I don't know how dill/pickle really works under the covers and no one else has offered their knowledge to investigate. |
Following up on this thread so as not to open another issue about pickling/serializing of wrapt decorators. For such a neatly implemented library it is quite the bummer that when trying to use multiprocessing/concurrent.futures/joblib with any wrapt decorated function the code fails with serialization errors. Given how common of a use case it is to parallelize code using processes (especially in data-intensive domains) - I am wondering if:
|
@marwan116 Since pickle is different from dill, with pickle being a part of Python, can you create a separate issue and also include in that a small reproducible example that could use to test the particular scenario you are wanting it to work on. |
Hey @GrahamDumpleton thank you for taking the time to respond. I will follow up with a small reproducible example shortly. |
@GrahamDumpleton - please refer to the following issue - (I don't see any guidelines on issue posting so please let me know if I missed anything) - #158 |
It would be extremely useful if wrapt functions were serializable in some way. Although pickle support is unlikely given that pickle doesn't even handle undecorated functions, I hoped the more capable dill may have been up to the task, as dill is capable of handling the decorator pattern in general:
unfortunately, with wrapt we get:
I would happily prepare a PR for wrapt which enabled support for dill serialization but I'm not too sure where to start. Certainly by inspection we can see that dill has failed to serialize the wrapt function in the first place:
I don't know if I would be better off bringing this to the dill author @mmckerns first or yourself, but if anyone can shed any light I would love to get this working.
The text was updated successfully, but these errors were encountered: