Update CometCallback to allow reusing of the running experiment

[Lothiraldan]

What does this PR do?

Update CometCallback to allow reusing of the running experiment.

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[amyeroberts]

Hi @Lothiraldan, thanks for opening a PR! Let us know when it's ready for review.

cc @muellerzr

[Lothiraldan]

Hi @amyeroberts, I wanted to open a PR in our fork for internal review first. Sorry about the noise.

I can close this one, reopen one in our fork or leave this one open and have the comments shared internally, let me know what would works best for you.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

@Lothiraldan Any of the options work for us. We can leave this open until your internal review has happened, this PR has been updated to reflect the review and then you can just ping us when it's ready :)

[Lothiraldan]

@amyeroberts Hi, this PR is now ready for a review. Sorry for the delay, the internal reviews highlighted some potential improvements that took a bit of time.

[muellerzr]

Thanks, this looks good to me. cc @amyeroberts for final review? :)

[LysandreJik]

Clean handling of the versioning. LGTM!