Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Logging in a callback does not work with multiple optimizers #5459

Closed
YannDubs opened this issue Jan 11, 2021 · 1 comment · Fixed by #5521
Closed

[BUG] Logging in a callback does not work with multiple optimizers #5459

YannDubs opened this issue Jan 11, 2021 · 1 comment · Fixed by #5521
Assignees
@carmocca
Labels
bug Something isn't working help wanted Open to be worked on priority: 0 High priority task

Comments

@YannDubs
Copy link

YannDubs commented Jan 11, 2021 •
edited
Loading

🐛 Bug

#5063 is not solved when logging in callbacks (@tchaton ). Specifically, self.log returns the following

File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/epoch_result_store.py", line 240, in auto_reduce_results_on_epoch_end
    opt_outputs = epoch_metrics[opt_idx]
KeyError: 0

when:

  • there are multiple optimizers
  • a callback calls pl_module.log

Please reproduce using the BoringModel

https://colab.research.google.com/drive/1JoHFDQx1vad6th2IhNSxL0m2n_1HzQro?usp=sharing

Environment

the bug appears in stable or master (see notebook)

Additional context

I'm using 2 optimizers and pl_bolts.callbacks.ssl_online.SSLOnlineEvaluator which does not work as it logs the loss in the callback. (Note that there are other issues with this callback and multiple optimizers #4955 )
@YannDubs YannDubs added bug Something isn't working help wanted Open to be worked on labels Jan 11, 2021
@tchaton tchaton added the priority: 0 High priority task label Jan 11, 2021
@tchaton tchaton self-assigned this Jan 11, 2021
@edenlightning edenlightning assigned Borda and unassigned tchaton and Borda Jan 11, 2021
@edenlightning
Copy link
Contributor

edenlightning commented Jan 11, 2021 •
edited by Borda
Loading

@akihironitta please take a look!

@carmocca carmocca self-assigned this Jan 14, 2021
@carmocca carmocca mentioned this issue Jan 15, 2021
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carmocca carmocca

Labels
bug Something isn't working help wanted Open to be worked on priority: 0 High priority task
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix logging on_train_batch_end in a callback with multiple optimizers
5 participants
@Borda @carmocca @tchaton @YannDubs @edenlightning