Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logging precision issue #9713

Closed
qqueing opened this issue Sep 27, 2021 · 1 comment · Fixed by #10076
Closed

logging precision issue #9713

qqueing opened this issue Sep 27, 2021 · 1 comment · Fixed by #10076
Labels
bug Something isn't working help wanted Open to be worked on

Comments

@qqueing
Copy link
Contributor

qqueing commented Sep 27, 2021

🐛 Bug

To Reproduce

#8887
#8930

This problem is logging precision.
If I use float32, logging process change my logging
(ex: .
validation_epoch_end(self, metrics)
'val_clk_sum': tensor(3211., device='cuda:0'),
-->
trainer.logged_metrics
'val_clk_sum': tensor(3210.9998, device='cuda:0'),
)

I know that if I use float 64 this problem doesn't happen.
But I don't want to use the trainer default of 64 because of logging.

Expected behavior

Above mentioned

Environment

  • PyTorch Lightning Version (e.g., 1.3.0):
  • PyTorch Version (e.g., 1.8)
  • Python version:
  • OS (e.g., Linux):
  • CUDA/cuDNN version:
  • GPU models and configuration:
  • How you installed PyTorch (conda, pip, source):
  • If compiling from source, the output of torch.__config__.show():
  • Any other relevant information:

Additional context

@qqueing qqueing added bug Something isn't working help wanted Open to be worked on labels Sep 27, 2021
@carmocca
Copy link
Contributor

Hi @qqueing

See my previous response: #8887 (comment). A way for us to reproduce the issue is necessary to fix it.

Until then, I'll have to close this issue again. If you can provide the code and your exact environment so that it's reproducible, then we will re-open and look into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Open to be worked on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants