Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix comet logger to log after train #892

Merged
merged 2 commits into from
Feb 22, 2020
Merged

Fix comet logger to log after train #892

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e9b3e8e
Fix comet logger to log after train
fdelrio89 Feb 18, 2020
0365ee1
Add clarifying comment to COmetLogger code
fdelrio89 Feb 21, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 32 additions & 7 deletions pytorch_lightning/loggers/comet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

try:
from comet_ml import Experiment as CometExperiment
from comet_ml import ExistingExperiment as CometExistingExperiment
from comet_ml import OfflineExperiment as CometOfflineExperiment
try:
from comet_ml.api import API
Expand All @@ -33,7 +34,8 @@ class CometLogger(LightningLoggerBase):
"""

def __init__(self, api_key=None, save_dir=None, workspace=None,
rest_api_key=None, project_name=None, experiment_name=None, **kwargs):
rest_api_key=None, project_name=None, experiment_name=None,
experiment_key=None, **kwargs):
r"""

Requires either an API Key (online mode) or a local directory path (offline mode)
Expand Down Expand Up @@ -100,6 +102,7 @@ def __init__(self, api_key=None, save_dir=None, workspace=None,

self.workspace = workspace
self.project_name = project_name
self.experiment_key = experiment_key
self._kwargs = kwargs

if rest_api_key is not None:
Expand Down Expand Up @@ -131,12 +134,22 @@ def experiment(self):
return self._experiment

if self.mode == "online":
self._experiment = CometExperiment(
api_key=self.api_key,
workspace=self.workspace,
project_name=self.project_name,
**self._kwargs
)
if self.experiment_key is None:
self._experiment = CometExperiment(
api_key=self.api_key,
workspace=self.workspace,
project_name=self.project_name,
**self._kwargs
)
self.experiment_key = self._experiment.get_key()
else:
self._experiment = CometExistingExperiment(
api_key=self.api_key,
workspace=self.workspace,
project_name=self.project_name,
previous_experiment=self.experiment_key,
**self._kwargs
)
else:
self._experiment = CometOfflineExperiment(
offline_directory=self.save_dir,
Expand All @@ -160,9 +173,21 @@ def log_metrics(self, metrics, step=None):

self.experiment.log_metrics(metrics, step=step)

def reset_experiment(self):
self._experiment = None

@rank_zero_only
def finalize(self, status):
r"""
When calling self.experiment.end(), that experiment won't log any more data to Comet. That's why, if you need
to log any more data you need to create an ExistingCometExperiment. For example, to log data when testing your
model after training, because when training is finalized CometLogger.finalize is called.

This happens automatically in the CometLogger.experiment property, when self._experiment is set to None
i.e. self.reset_experiment().
"""
self.experiment.end()
self.reset_experiment()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the need for setting self._experiment to None?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When calling self.experiment.end(), that experiment won't log any more data to Comet (that was issue #760 about). That's why, if you need to log any more data you need to create an ExistingCometExperiment. For example, to log data when testing your model after training, because when training is finalized CometLogger.finalize is called.

With this PR, this happens automatically in the CometLogger.experiment property, when self._experiment is set to None.

Tell me if anything is not clear yet :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to write this your explanation also to code as a comment for future... 🤖


@property
def name(self):
Expand Down