Cannot load class with metaclass thats using dynamic classes

[verybadsoldier]

These are my first steps with dill and I am trying to pickle a calculation result of the backtrader framework which is a stock trading framework (https://www.backtrader.com/).

Using dill did not work out of the box. It failed on trying to unpickle the object. backtrader makes heavy use of metaclasses and classes that are generated at runtime so I guess it is a difficult case for dill.

From the backtrader framework I extracted and copy/pasted some code to build a minimal example to show the problem. It fails on dill.load(open(filename, 'rb')):

import sys
import dill
from collections import OrderedDict


class MetaParams(type):
    def __new__(meta, name, bases, dct):
        # Remove params from class definition to avod inheritance
        # (and hence "repetition")
        newparams = dct.pop('params', ())

        # Create the new class - this pulls predefined "params"
        cls = super(MetaParams, meta).__new__(meta, name, bases, dct)

        # Subclass and store the newly derived params class
        cls.params = MetaParams._derive(name, newparams)
        return cls

    @classmethod
    def _derive(cls, name, info):
        clsinfo = OrderedDict()
        clsinfo.update(info)

        clsmodule = sys.modules['__main__']
        newclsname = "MyDynamicClass"

        newcls = type(newclsname, (cls,), {})
        setattr(newcls, '_getpairs', classmethod(lambda cls: clsinfo.copy()))
        setattr(clsmodule, newclsname, newcls)
        newcls.__module__ = '__main__'

        return newcls


class LineRoot(object, metaclass=MetaParams):
    params = dict(myParam='myValue')


filename = "C:\\archive\\s3.dill"

dill.dump(LineRoot, open(filename, 'wb'))

dill.load(open(filename, 'rb'))

This gives me this exception:

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2017.3\helpers\pydev\pydevd.py", line 1668, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm 2017.3\helpers\pydev\pydevd.py", line 1662, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm 2017.3\helpers\pydev\pydevd.py", line 1072, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2017.3\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/vbs/Projects/dilltest/strat.py", line 42, in <module>
    dill.load(open(filename, 'rb'))
  File "C:\Users\vbs\Miniconda3\envs\qrawler\lib\site-packages\dill\dill.py", line 288, in load
    obj = pik.load()
  File "C:\Users\vbs\Miniconda3\envs\qrawler\lib\site-packages\dill\dill.py", line 549, in _create_type
    return typeobj(*args)
  File "C:/Users/vbs/Projects/dilltest/strat.py", line 16, in __new__
    cls.params = MetaParams._derive(name, newparams)
  File "C:/Users/vbs/Projects/dilltest/strat.py", line 22, in _derive
    clsinfo.update(info)
TypeError: 'type' object is not iterable

My interpretation of what is happening (while I am sure you are much better at understanding it as I am):
The class LineRoot has to be pickled. It gets instantiated by the metaclass MetaParams. On normal run, the parameter dct in this method:

class MetaParams(type):
    def __new__(meta, name, bases, dct):

is a dict which contains the entry params which again is a dict. Based on params the method _derive creates a new class which is assigned as the new value of the class variable params.

Now when running the unpickle process then the params entry in the dct already is the dynamic class that should be created later. The code does not expect it to be that and fails.

---

In the code there are several places where this pattern is used. Is there any chance to get it working with dill?
Big thanks in advance!

---

Oh this is my environment:
Python 3.6.3 :: Anaconda, Inc.

dill 0.2.7.1 py36hf552773_0

Windows 10

[anivegesana]

This was from a long time ago, but it is kind of relevant. Right now, __new__ gets called once during the initial creation of the class, and another time when the pickle file is unpickled. #381 is a more minimal example of this. Because of this, functions that initialize classes (namely __prepare__, __new__, and __init_subclass__) must be reentrant. A very simple change to the code will solve this issue (on branches that have #468): wrap the __new__ function in a if statement that tests if the __new__ function has already been entered once before.

    def __new__(meta, name, bases, dct):
        if '_new_entered' not in dct:

            # Remove params from class definition to avod inheritance
            # (and hence "repetition")
            newparams = dct.pop('params', ())
            
            # Create the new class - this pulls predefined "params"
            cls = super(MetaParams, meta).__new__(meta, name, bases, dct)
            
            # Subclass and store the newly derived params class
            cls.params = MetaParams._derive(name, newparams)

            cls._new_entered = True
            return cls
        else:
            return type.__new__(meta, name, bases, dct)

[mmckerns]

@anivegesana: nice workaround -- of course people shouldn't have to write that many lines of specialized code for a workaround. It seems, however, that you could pack it into a decorator. It would be new to dill to provide a function that helps serialization, as opposed to the current "hands-off" approach.

[anivegesana]

Yep. I was going to suggest that there would be a setting to turn off class initialization, but I haven't yet found time to see if this was possible.