Archive and lambda functions

[vascotenner]

I try to add arrows to my plots (see #34). The archive cannot handle lambda functions:

hv.archive.auto()

def add_arrow(plot, element, xy, xytext, arrowprops=dict(arrowstyle="->", connectionstyle="arc3")):
    axis = plot.handles['axis']
    axis.annotate("", xy=xy, xycoords='data',
          xytext=xytext, textcoords='data',
          arrowprops=arrowprops)

%%opts Curve [final_hooks=[lambda *args: add_arrow(xy=(0,0), xytext=(1,0), *args)]]
hv.Curve([1,1])

PicklingError: Can't pickle <function <lambda> at 0x7f757e56c938>: it's not found as __main__.<lambda>

[jbednar]

Without special magic, lambda functions cannot be pickled in Python , so you'd need to (a) make a named function in a well-defined location (an importable module) for your hook, and (b) you'd need to make sure that this module is available for import later when you read the archive.

We've made hacks to pickle the bytecode for lambdas in some projects, but that's not a robust approach in general...

[vascotenner]

Is there a way to not even try to pickle lambda functions while archiving the notebook? Most of the time, I use the archive function to have:

• A fast loading overview of the notebook contents
• SVG export of all images

[philippjfr]

Both cloudpickle and dill promise to work as a dropin replacement for pickle allowing lambdas. Might be worth trying if it just works, then we could support it by parameterizing the pickle library that is used.

[jbednar]

It might be handy to provide a parameter to only warn on unpicklable objects, rather than failing? Not sure I'd want to commit to an external library, since pickling lambdas probably isn't ever going to be that reliable, even if it's supported. Both cloudpickle and dill focus on shipping objects around various nodes for computation, rather than archiving as we are doing here, and so they don't need to worry as much about long-term persistence of the various objects.

[jlstevens]

It might be handy to provide a parameter to only warn on unpicklable objects, rather than failing?

I agree! I'll assign myself this issue in order to make sure warnings are issued instead of failures.

In addition, I also wrote the following reply during the brief GitHub outage:

Your example sets a lambda in a plot option - we could consider trying to strip out lambdas when pickling but then the plot would change when unpickled.

Are you using the pickles at all? From the sound of it, you might just want the SVG export. This might be an argument for removing the pickler by default:

class NotebookArchive(FileArchive):
    """
    FileArchive that can automatically capture notebook data via the
    display hooks and automatically adds a notebook HTML snapshot to
    the archive upon export.
    """
    exporters = param.List(default=[Pickler])

If you only want the SVG output, you can try removing the Pickler from hv.archive.exporters.

[jbednar]

Sounds good. I guess for my own purposes I'd want the "pickle_failures_as_warnings" flag to be on by default, since I'd rather have a run succeed but possibly have some un-unpicklable state than to have the whole thing fail for a reason that usually doesn't matter. But different people might have different needs, and so might want to re-enable the current behavior (e.g. if they have set up an automated document-building workflow crucially relying on the full objects to be unpicklable).

[jbednar]

And of course those users might want dill or cloudpickle support, as Philipp suggests, because the pickles would then be for enabling a specific workflow rather than for long-time archiving. But as I don't think any of us are in that category, maybe best to wait on such super-pickle support until it's clear that someone actually needs such potentially problematic functionality.