ConsoleRenderer and TimeStamper can't be serialized

[bonheml]

These two object should be able to be pickled.
For now, they can't be used with multiprocessing which is common use case.

Here is a script to recreate the bug:

import structlog
import pickle

if __name__=="__main__":
    try:
        pickle.dumps(structlog.dev.ConsoleRenderer(colors=False))
    except Exception as e:
        print(e)
    try:
        pickle.dumps(structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M:%S"))
    except Exception as e:
        print(e)

It leads to the following stackTrace

/usr/bin/python3.5 /home/lisa/Dev/log_sanbox/main_logger_structlog.py
Can't pickle local object 'ConsoleRenderer.__init__.<locals>._repr'
Can't pickle <class 'structlog.processors.TimeStamper'>: it's not the same object as structlog.processors.TimeStamper

[hynek]

Hmmm, the problem with those two processors is that that they write JIT optimal versions of themselves for each possible configuration.

Do you happen to know more about pickling whether we could work around that somehow?

[bonheml]

Thanks for your answer. I'm not sure to understand correctly your question about pickling but I'll try to detail more my use case.

I have a program which use python multiprocessing and during spawn of processes, my logger can't be pickled leading to an error.

I work with python 3.5 on Fedora

[MarcBoissonneault]

Any new on this issue?
The ConsoleRenderer still seems to be not serializable:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.7/copy.py:180: in deepcopy
    y = _reconstruct(x, memo, *rv)
/usr/lib/python3.7/copy.py:280: in _reconstruct
    state = deepcopy(state, memo)
/usr/lib/python3.7/copy.py:150: in deepcopy
    y = copier(x, memo)
/usr/lib/python3.7/copy.py:240: in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
/usr/lib/python3.7/copy.py:161: in deepcopy
    y = copier(memo)
/usr/local/lib/python3.7/dist-packages/structlog/_base.py:176: in _proxy_to_logger
    args, kw = self._process_event(method_name, event, event_kw)
/usr/local/lib/python3.7/dist-packages/structlog/_base.py:136: in _process_event
    event_dict = proc(self._logger, method_name, event_dict)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <structlog.dev.ConsoleRenderer object at 0x7f99a35ae208>
_ = <PrintLogger(file=<_pytest.capture.EncodedFile object at 0x7f9a31086748>)>
__ = '__deepcopy__', event_dict = {}

    def __call__(self, _, __, event_dict):
        sio = StringIO()
    
        ts = event_dict.pop("timestamp", None)
        if ts is not None:
            sio.write(
                # can be a number if timestamp is UNIXy
                self._styles.timestamp + str(ts) + self._styles.reset + " "
            )
        level = event_dict.pop("level", None)
        if level is not None:
            sio.write(
                "[" + self._level_to_color[level] +
                _pad(level, self._longest_level) +
                self._styles.reset + "] "
            )
    
        event = event_dict.pop("event")
        if event_dict:
            event = _pad(event, self._pad_event) + self._styles.reset + " "
        else:
>           event += self._styles.reset
E           TypeError: unsupported operand type(s) for +=: 'dict' and 'str'

/usr/local/lib/python3.7/dist-packages/structlog/dev.py:191: TypeError

[creslinux]

Ive ran into this. Was there any resolution ?
I seem tied to a single core with structlog in my application, this isn't workable as i need to bring 10's of cores online to crunch a dataframe many ways.

[berlinguyinca]

could this be fixed at this stage? It's been a couple of years now

[hynek]

I would love to fix it but it's not as simple without a notable performance regression so it's not on the top of my todo list. :(

[berlinguyinca]

could we in this case have an alternative slower version, just to be able to use it somehow?

…

On Wed, May 8, 2019 at 11:19 AM Hynek Schlawack ***@***.***> wrote: I would love to fix it but it's not as simple without a notable performance regression so it's not on the top of my todo list. :( — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#126 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAD73G3HNIBYVUG34SPTGLPUMKTTANCNFSM4DNZXIQQ> .

-- ------------------------------------------------------------ Lead Developer - Fiehnlab, UC Davis gert wohlgemuth work: http://fiehnlab.ucdavis.edu/staff/wohlgemuth phone: 530 665 9477 coding blog: http://codingandmore.blogspot.com linkedin: http://www.linkedin.com/profile/view?id=28611299&trk=tab_pro

[hynek]

If that's all you want, you can write your own. If you implement only what you need, it becomes three lines of code or so:

structlog/src/structlog/processors.py

Lines 272 to 274 in 55293fb

	def stamper(self, _, __, event_dict):
	event_dict[key] = now_method().strftime(fmt)
	return event_dict

If someone want to help out and fix ConsoleRenderer: I do not care about its performance so it can be implemented however.

[hynek]

I've just made TimeStamper, ConsoleRenderer, the config proxy you get from get_logger, PrintLogger and the generic BoundLogger pickleable.

Is anything missing or can I release soon? It would be great if you could check your use cases against master. The default config should definitely work.

[berlinguyinca]

Thanks this sounds great

…

On Sat, Sep 21, 2019, 5:02 AM Hynek Schlawack ***@***.***> wrote: I've just made TimeStamper, ConsoleRenderer, the config proxy you get from get_logger, PrintLogger and the generic BoundLogger pickleable. Is anything missing or can I release soon? It would be great if you could check your use cases against master. The default config should definitely work. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#126?email_source=notifications&email_token=AAAD73CGPZXWM2RUOMLJHH3QKYEOVA5CNFSM4DNZXIQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7IQPEI#issuecomment-533792657>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAD73BTEMNN3H7VXFOWKVTQKYEOVANCNFSM4DNZXIQQ> .

[berlinguyinca]

ok works much better! And the error is a lot clearer now! On Sat, Sep 21, 2019 at 4:10 PM Gert Wohlgemuth <berlinguyinca@gmail.com> wrote:

…

Thanks this sounds great On Sat, Sep 21, 2019, 5:02 AM Hynek Schlawack ***@***.***> wrote: > I've just made TimeStamper, ConsoleRenderer, the config proxy you get > from get_logger, PrintLogger and the generic BoundLogger pickleable. > > Is anything missing or can I release soon? It would be great if you could > check your use cases against master. The default config should definitely > work. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#126?email_source=notifications&email_token=AAAD73CGPZXWM2RUOMLJHH3QKYEOVA5CNFSM4DNZXIQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7IQPEI#issuecomment-533792657>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAAD73BTEMNN3H7VXFOWKVTQKYEOVANCNFSM4DNZXIQQ> > . >

-- ------------------------------------------------------------ Lead Developer - Fiehnlab, UC Davis gert wohlgemuth work: http://fiehnlab.ucdavis.edu/staff/wohlgemuth phone: 530 665 9477 coding blog: http://codingandmore.blogspot.com linkedin: http://www.linkedin.com/profile/view?id=28611299&trk=tab_pro

[hynek]

OK I’m closing this then. Please report if y’all have problems with any other class.