Update the Neptune integration #906

twolodzko · 2022-10-11T13:56:33Z

Neptune integration was updated to work with a recent version of the Neptune client.
Additional logging was added: model summary, configuration (learning rate, optimizer, etc), event_lr, and epoch duration.
Tests were altered to match the current API of the client library and improved when possible.

Minor language fixes

BenjaminBossan

Thank you very much for the update. I played around with the new neptune logger and experiment tracking site and was impressed. It's very useful and there are nice additions since last time I checked.

I have a few code comments below, please take a look. On top of those, I have some more general comments:

Versions

I saw that you have some code that works with neptune-client>=1.0, which I didn't find released on PyPI. So I assume it's for forward compatibility for a yet to be released version. Big 👍.

That said, I ran into some issues with certain Neptune versions (see one of my code comments). So I think that there could still be an issue with supporting all versions >=0.9.0. I would thus like to ask you to check again if it works with all intended versions. Maybe some older ones can be dropped? At the end of the day, you know best which versions should still be supported and how.

Pickling

A minor inconvenience I found is that the NeptuneLogger apparently can't be pickled:

AttributeError: Can't pickle local object 'NeptuneAuthenticator.__init__.<locals>.session_factory'

Therefore, a user would not be able to save the whole skorch net on Neptune. If there is an easy way to fix this by doing something with NeptuneLogger.get_state, that would be nice, but I'm not sure. On the neptune-client side, these lines seem to be the offender:

https://github.com/neptune-ai/neptune-client/blob/81fcf25b204331ae85b67e19afc78be69ddf3f84/src/neptune/oauth.py#L98-L126

Since a reference to a local function is stored, pickle fails. So if you ever want to fix this, you would have to move this definition to the module level.

Tests

While running the tests, I got this message multiple times:

Error in atexit._run_exitfuncs:

Not sure what it means and if it can cause any trouble.

skorch/callbacks/logging.py

BenjaminBossan · 2022-10-14T09:52:39Z

@twolodzko Let me know when you are finished and need another review round.

twolodzko · 2022-10-17T08:31:28Z

That said, I ran into some issues with certain Neptune versions (see one of my code comments).

@BenjaminBossan I changed the requirements to a higher version.

twolodzko · 2022-10-17T09:32:48Z

Pickling

A minor inconvenience I found is that the NeptuneLogger apparently can't be pickled:

AttributeError: Can't pickle local object 'NeptuneAuthenticator.__init__.<locals>.session_factory'
[...]

Would this be blocking? It seems that this is something we should rather fix in the Neptune client library that is unpickable than this logger. Would it be ok if we merged the logger like this, but in the meantime we would figure out on our side how to make the client pickle-friendly?

Tests

While running the tests, I got this message multiple times:

Error in atexit._run_exitfuncs:

Not sure what it means and if it can cause any trouble.

This is not a problem, they can be ignored. The errors raise because we are using async calls and pytest does not handle closing them well. If I understand correctly, the tests have passed but you just saw this in the logs?

twolodzko · 2022-10-17T09:38:21Z

@BenjaminBossan I think I addressed all of your comments.

BenjaminBossan

Thank you a lot for making the changes. Everything looks very good now. I only have a minor change request.

Would this be blocking? It seems that this is something we should rather fix in the Neptune client library that is unpickable than this logger. Would it be ok if we merged the logger like this, but in the meantime we would figure out on our side how to make the client pickle-friendly?

No, not an issue. As a skorch user, I can still use the Neptune logger and then purge it from the callbacks to be able to pickle the model. Most often, the logger would not be required after loading anyway, since at that point, most users are only interested in predicting with the model.

If you do make a change in Neptune that ends up enabling pickle, please let us know and we can add a unit test to check this.

skorch/tests/callbacks/test_logging.py

BenjaminBossan · 2022-10-18T14:43:55Z

@twolodzko Great, the "atexit" messages disappeared, thanks for fixing that.

I have a suggestion to make regarding the _extract_name_parts function. How about removing it and then inside of _log_metric, change the first lines to:

    def _log_metric(self, name, logs, batch):
        # split e.g. train_loss into train and loss if possible
        kind, _, key = name.partition('_')

        if not key:
            key = 'epoch_duration' if kind == 'dur' else kind
            self._metric_logger[key].log(logs[name])
        else:
            ...

Would that achieve the same goal? If so, I would prefer it, it seems easier to understand.

shnela · 2022-10-18T16:46:16Z

skorch/callbacks/logging.py

-            self.experiment.stop()
+            self.run.stop()
+
+    def _log_metric(self, name, logs, batch):


I'm a little puzzled about this function.
Are those shortcuts like dur or valid some internal skorch strings which we're replacing to increase the readability of our experiment?
Could we add some comment here?

@shnela yes. Skorch logs dur dor duration of the epoch and several metrics like valid_loss or train_loss, where we extend valid to validation for readability.

twolodzko · 2022-10-19T07:38:15Z

I have a suggestion to make regarding the _extract_name_parts function. [...]

@BenjaminBossan Nice idea. Done.

BenjaminBossan

Thanks a lot @twolodzko @normandy7 @shnela. Very nice additions and well thought out changes. I made some minor edits to the CHANGES.md entry, I hope you don't mind.

twolodzko · 2022-10-19T09:05:49Z

@BenjaminBossan sure, thanks.

Herudaio · 2022-11-04T09:18:53Z

@BenjaminBossan is there any rough ETA on when a release including these changes may happen? :)

BenjaminBossan · 2022-11-04T09:29:28Z

No, not really, we typically just release when enough changes have accumulated and the last release is very recent, so it'll probably be a couple of months. If it's important for you to have an earlier release, we could consider a patch-level release.

Herudaio · 2022-11-09T13:24:55Z

Thanks for the information.

If it's important for you to have an earlier release, we could consider a patch-level release.

@BenjaminBossan that would be awesome, if there's such a possibility! :)

BenjaminBossan · 2022-11-10T09:58:17Z

@Herudaio Yes, I think we can do that, hopefully not later than end of next week.

BenjaminBossan · 2022-11-18T12:53:48Z

@Herudaio We just released v0.12.1 of skorch with the updated logger.

twolodzko and others added 9 commits October 11, 2022 11:44

Neptune integration update

da055c8

Merge remote-tracking branch 'upstream/master'

69bbb16

Mentione base_namespace in docstring

a30d20f

Rename experiment to run

7c81ad3

Minor language fixes

de33f84

Merge pull request #1 from normandy7/patch-1

40823d9

Minor language fixes

Minor improvements in docs

d54a7d1

Update CHANGELOG

9624d59

Rename dur to epoch_duration

b36d23a

BenjaminBossan requested changes Oct 12, 2022

View reviewed changes

twolodzko added 2 commits October 13, 2022 11:22

Fixes after review

2e1df82

Update link with an example for Neptune integration

014d03d

Use higher version of neptune client

8e08269

Add TODO

36eaf15

BenjaminBossan requested changes Oct 17, 2022

View reviewed changes

skorch/tests/callbacks/test_logging.py Show resolved Hide resolved

twolodzko added 3 commits October 18, 2022 15:21

Fix noisy warnings in unit tests

9ee7df5

Extract teh _extract_name_parts method

08b09c5

Simplify _extract_name_parts

103248a

shnela reviewed Oct 18, 2022

View reviewed changes

Simplify extraction of name parts

722d462

shnela approved these changes Oct 19, 2022

View reviewed changes

Small update to CHANGES.md entry

183819a

BenjaminBossan approved these changes Oct 19, 2022

View reviewed changes

BenjaminBossan merged commit 647702b into skorch-dev:master Oct 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the Neptune integration #906

Update the Neptune integration #906

twolodzko commented Oct 11, 2022 •

edited

Loading

BenjaminBossan left a comment

BenjaminBossan commented Oct 14, 2022

twolodzko commented Oct 17, 2022

twolodzko commented Oct 17, 2022

Pickling

Tests

twolodzko commented Oct 17, 2022

BenjaminBossan left a comment

BenjaminBossan commented Oct 18, 2022

shnela Oct 18, 2022

twolodzko Oct 19, 2022 •

edited

Loading

twolodzko commented Oct 19, 2022 •

edited

Loading

BenjaminBossan left a comment •

edited

Loading

twolodzko commented Oct 19, 2022

Herudaio commented Nov 4, 2022

BenjaminBossan commented Nov 4, 2022

Herudaio commented Nov 9, 2022

BenjaminBossan commented Nov 10, 2022

BenjaminBossan commented Nov 18, 2022

Update the Neptune integration #906

Update the Neptune integration #906

Conversation

twolodzko commented Oct 11, 2022 • edited Loading

BenjaminBossan left a comment

Choose a reason for hiding this comment

Versions

Pickling

Tests

BenjaminBossan commented Oct 14, 2022

twolodzko commented Oct 17, 2022

twolodzko commented Oct 17, 2022

Pickling

Tests

twolodzko commented Oct 17, 2022

BenjaminBossan left a comment

Choose a reason for hiding this comment

BenjaminBossan commented Oct 18, 2022

shnela Oct 18, 2022

Choose a reason for hiding this comment

twolodzko Oct 19, 2022 • edited Loading

Choose a reason for hiding this comment

twolodzko commented Oct 19, 2022 • edited Loading

BenjaminBossan left a comment • edited Loading

Choose a reason for hiding this comment

twolodzko commented Oct 19, 2022

Herudaio commented Nov 4, 2022

BenjaminBossan commented Nov 4, 2022

Herudaio commented Nov 9, 2022

BenjaminBossan commented Nov 10, 2022

BenjaminBossan commented Nov 18, 2022

twolodzko commented Oct 11, 2022 •

edited

Loading

twolodzko Oct 19, 2022 •

edited

Loading

twolodzko commented Oct 19, 2022 •

edited

Loading

BenjaminBossan left a comment •

edited

Loading