WandbLogger does not mark uploaded model as 'artifact'

[kyoungrok0517]

🐛 Bug

I'm using WandbLogger with the latest pytorch-lightning==1.0.8. It seems like trained checkpoint is treated as mere file not a model artifact, even I turned on log_model=True. It's much convenient to use model artifact from other script so I hope that is done by pytorch-lightning automatically.

Environment

* CUDA:
        - GPU:
                - GeForce RTX 3090
        - available:         True
        - version:           11.0
* Packages:
        - numpy:             1.19.2
        - pyTorch_debug:     True
        - pyTorch_version:   1.7.0
        - pytorch-lightning: 1.0.8
        - tensorboard:       2.3.0
        - tqdm:              4.51.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - ELF
        - processor:         x86_64
        - python:            3.8.5
        - version:           #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020

[Borda]

@borisdayma mind have look?

[borisdayma]

This is a great idea!
Right now, wandb uploads the saved model folder at the end of the run, which may include one or several models (when saving checkpoints).

We could upload artifacts at the end of the run or as the model trains.
I think it may make more sense to upload at the end of training to avoid uploading unnecessarily too many times the model, for example if we are only interested in the final version or only the top 5 best models.

Finally, I think each artifact name should be related to the run id so if we want to supersede a specific artifact, we will need to use the same run id (knowing that W&B still let you use any number of alias latest, best, etc on each artifact).

Let me know if you have any comments before I try to implement it.

[kyoungrok0517]

This is a great idea!
Right now, wandb uploads the saved model folder at the end of the run, which may include one or several models (when saving checkpoints).

We could upload artifacts at the end of the run or as the model trains.
I think it may make more sense to upload at the end of training to avoid uploading unnecessarily too many times the model, for example if we are only interested in the final version or only the top 5 best models.

Finally, I think each artifact name should be related to the run id so if we want to supersede a specific artifact, we will need to use the same run id (knowing that W&B still let you use any number of alias latest, best, etc on each artifact).

Let me know if you have any comments before I try to implement it.

Thanks for the response. I think following the checkpoint settings (e.g. preserve best top-k) would be nice, as you've mentioned in your comment. Then we can try the models in the other script using wandb API (e.g. automatic download).

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[borisdayma]

Keep issue active

[tchaton]

Hey @borisdayma,

Did you start to work on it ?

Best,
T.C

[borisdayma]

I didn't have the time yet but it's definitely on my TODO list.
I have a PR ongoing and I prefer to wait for it to be merged before.

[borisdayma]

I am now working on it and I just realized that #5537 introduced a change in behavior at this line.

Models were supposed to be saved in the W&B run folder only when log_model=True (since those files may be automatically uploaded at the end of a run).

With artifacts, we can let PL save the files directly where it wants and upload models as artifacts separately.

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[borisdayma]

PR has now been merged so this issue should be closed!