Comet Logging and Visualization Integration #9232

DN6 · 2022-08-31T06:48:23Z

This PR:

Adds support for logging metrics, hyperparameters, model predictions, checkpoints, and datasets from the YOLOv5 train.py script.
Adds supports for resuming a training run from an Existing Comet Experiment
Adds support for downloading checkpoints from a Comet Experiment
Adds support for downloading a dataset from Comet Artifacts downloading
Adds support for using the Comet Hyperparameter Optimizer with the YOLOv5 train.py script.
Adds the relevant section on how to use Comet with YOLOv5 to tutorials.ipynb
Adds a README guide for the integration under utils/loggers/comet to explain the features of the integration

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhancements to YOLOv5 with the integration of Comet ML for logging and visualization.

📊 Key Changes

Removed the "Environments" section and logos from the README for a cleaner look.
Added a new "Integrations" image to README to visualize integration partners.
Added code to enable Comet ML for tracking experiments.
Included callbacks modification to support custom logging in training loops.
Created a dedicated README for Comet ML to facilitate user understanding.
Added Comet ML to the logger integrations in the code, alongside other platforms like Weights & Biases.
Implemented documentation and utility scripts for Comet ML integration.

🎯 Purpose & Impact

Streamlines YOLOv5's README and enhances visual appeal 🎨.
Allows users to track and visualize their YOLOv5 model training metrics in real-time using Comet ML 📈.
Gives users new tools for experiment management and collaboration, thus boosting productivity 🚀.
Strengthens the ecosystem of integrations, making YOLOv5 more versatile and user-friendly 🌐.

DN6 · 2022-08-31T14:02:43Z

@glenn-jocher @AyushExel Here is the new PR without any external formatting.

AyushExel · 2022-09-01T04:17:23Z

@DN6 thanks for opening another PR. We're trying to limit any changes to core files except utils/loggers for logging use case and we'll do the same for existing loggers later this year.
Let's try to finalize changes that are already independent of core file changes. I think we should finalize these first:

Adds support for logging metrics, hyperparameters, model predictions, checkpoints, and datasets from the YOLOv5 train.py script.
Adds support for downloading a dataset from Comet Artifacts downloading
Adds support for using the Comet Hyperparameter Optimizer with the YOLOv5 train.py script.
Adds the relevant section on how to use Comet with YOLOv5 to tutorials.ipynb
Adds a README guide for the integration under utils/loggers/comet to explain the features of the integration

DN6 · 2022-09-01T07:25:16Z

Hi @AyushExel.

Regarding the first point

[ ] Adds support for logging metrics, hyperparameters, model predictions, checkpoints, and datasets from the YOLOv5 train.py script.

Logging model predictions (logging the Confusion Matrix, Images, Predictions and GT-Labels) requires updating the callback args in val.py. Is there any alternative way of getting this information into utils/loggers?

We can definitely prioritize the features you listed first, I'd just like to know how to proceed on the remaining ones, since they are quite useful. Are there any specific changes in the core files that are an issue?

AyushExel · 2022-09-01T07:33:20Z

@DN6 confusion matrix and other results are being logged to all loggers in this function https://github.com/ultralytics/yolov5/blob/master/utils/loggers/__init__.py#L201
It should not require any changes in any other function

DN6 · 2022-09-01T09:27:39Z

We are logging an interactive confusion matrix, which requires accessing the matrix object from inside YOLO.

AyushExel · 2022-09-05T11:28:46Z

@DN6 I want to make another change. The logic should be this:

If bbox_interval is -1, then we can set the interval to epochs//10
else use whatever is provided
What do you think?

AyushExel · 2022-09-05T11:47:18Z

can you take a look at my dashboard https://www.comet.com/ayushexel/yolov5/4ddf2a99b0d74b629ef4b02a053c5946?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step
I set bbox_interval to 1 but I don't see the images panel in the dashboard. The images are available in the graphics section

DN6 · 2022-09-05T12:34:12Z

Hi @AyushExel.

I can't see your project, you'll have to change the project permissions (manage -> Project Visibility -> Change to Public -> Save Changes)

The Image Panel has to be added in by the user. It doesn't auto-populate. In the Panel dashboard page, go to Add -> New Panel -> Public Panels -> Bounding box (It should show up if you search bounding box)

Regarding bbox_interval, what is the behavior if epochs is less than 10?

AyushExel · 2022-09-05T12:54:57Z

@DN6 we have 2 options if epochs is leass than 10.. Either log for all steps or log for none. Here's the flow:

if bbox_interval == -1:
     bbox_interval = epochs//10 if epochs < 10 else 1

DN6 · 2022-09-05T13:11:24Z

I see. So image logging will always be enabled? Will there be any way to disable it?

Also, is the bbox_interval supposed to correspond to an interval for every batch (i.e. log images from every Nth batch of data) or an interval for the epoch (log all validation images every Nth epoch). I assumed logging all the images from an epoch might result in a very large number of images being logged.

AyushExel · 2022-09-05T15:41:49Z

@DN6 no.. Ideally the same images should be logged for each step because then its easier to see the progress of the model.

DN6 · 2022-09-06T05:16:23Z

Got it. I'm good with that change then. I'll make the relevant change shortly.

DN6 · 2022-09-06T08:03:05Z

@AyushExel Changes for bbox_interval have been pushed up.

DN6 · 2022-09-07T05:46:22Z

Hi @AyushExel. Just following up. When do you anticipate we can merge this PR?

AyushExel · 2022-09-07T05:47:47Z

@DN6 soon. I've only taken a look at the core files and those look good to me now.
Waiting on @glenn-jocher to test and merge

glenn-jocher · 2022-09-07T12:15:36Z

@DN6 @AyushExel got it, thanks guys, taking a look today.

glenn-jocher · 2022-09-07T14:21:59Z

@DN6 @AyushExel cleaned up the notebook changes. Before was +1970 - 591, now +1343 - 22.

glenn-jocher · 2022-09-07T14:52:10Z

@DN6 do you have a square Comet logo PNG? All I can find online is rectangle shape, but I need a square with the right empty area top and bottom to match the other logos, i.e.:

EDIT: Nevermind, I'm all good.

for more information, see https://pre-commit.ci

glenn-jocher · 2022-09-07T15:28:10Z

Looking good guys. I updated the README table and graphic as well.

glenn-jocher · 2022-09-07T15:28:59Z

@DN6 @AyushExel PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

DN6 · 2022-09-07T15:54:15Z

Thank you for merging and adding the logo @glenn-jocher!

Can we also link to the Comet tutorial README from the tutorials section in the main YOLOv5 README ? (I can add this as a separate PR if needed)

And could we also make an update to the YOLOV5 wiki to include the Comet section mentioned in the tutorials.ipynb notebook please.

glenn-jocher · 2022-09-07T16:28:06Z

@DN6 perfect, yes please submit README PR and I'll update wiki.

* add comet to logger interface * add comet logger * add support for updated parameters * clean up offline logger creation * update callback args for comet logger * add comet optimizer * add optimizer config * add comet README * update tutorial notebook with Comet section * add option to log class level metrics * add support for class level metrics and confusion matrix * handle errors when adding files to artifacts * fix typo * clean resume workflow * updates for HPO * update comet README * fix typo in comet README * update code snippets in comet README * update comet links in tutorial * updated links * change optimizer batch size param and update comet README image * update comet section in tutorial * use prexisting cmd line flags to configure logger * update artifact upload/download flow * remove come remove comet logger specific cmd line args * move downloading weights into comet logger code * remove extra argparse * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * change checkpoint logging flow to follow offline logger * update resume flow * add comet logger to remote dataset property * update cmd line args in hpo * set types for integer/float env variables * update README * fix typo in README * default to always logging model predictions * Update tutorial.ipynb * Update train.py * Add Comet to Integrations table * Update README.md * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: Ayush Chaurasia <ayush.chaurarsia@gmail.com> Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

DN6 added 24 commits August 30, 2022 05:17

add comet to logger interface

6ec6e65

add comet logger

57ce76c

add support for updated parameters

fa6610e

clean up offline logger creation

1fce3df

update callback args for comet logger

751dcc3

add comet optimizer

2e10454

add optimizer config

d08e10b

add comet README

616e2db

Merge upstream changes from 'master' into feature-comet-integration

d3d01da

update tutorial notebook with Comet section

680ef8b

add option to log class level metrics

949cb44

add support for class level metrics and confusion matrix

aac31c2

Merge upstream 'master' into feature-comet-integration

33ea49d

handle errors when adding files to artifacts

1f0eb71

fix typo

24842bb

clean resume workflow

7a87143

updates for HPO

501ba8a

update comet README

9ae182e

fix typo in comet README

177ac97

update code snippets in comet README

e1c09d8

update comet links in tutorial

eb12ea5

updated links

d2f931f

change optimizer batch size param and update comet README image

5f959df

update comet section in tutorial

ee7b82c

Merge branch 'master' into feature-comet-integration

a48ee2c

fix typo in README

937bb7c

default to always logging model predictions

f273392

Merge branch 'master' into feature-comet-integration

83b9a4f

Merge branch 'master' into feature-comet-integration

288d925

Update tutorial.ipynb

0f97d4a

Update train.py

b7a210b

glenn-jocher and others added 3 commits September 7, 2022 18:09

Add Comet to Integrations table

c01efb6

Update README.md

c976c64

[pre-commit.ci] auto fixes from pre-commit.com hooks

1b33dd6

for more information, see https://pre-commit.ci

glenn-jocher changed the title ~~Comet Integration~~ Comet Logging and Visualization Integration Sep 7, 2022

glenn-jocher merged commit 903b239 into ultralytics:master Sep 7, 2022

Hojland mentioned this pull request Oct 17, 2022

feat/bump Go-Autonomous/yolov5#15

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comet Logging and Visualization Integration #9232

Comet Logging and Visualization Integration #9232

DN6 commented Aug 31, 2022 •

edited by UltralyticsAssistant

Loading

DN6 commented Aug 31, 2022 •

edited

Loading

AyushExel commented Sep 1, 2022

DN6 commented Sep 1, 2022

AyushExel commented Sep 1, 2022

DN6 commented Sep 1, 2022 •

edited

Loading

AyushExel commented Sep 5, 2022 •

edited

Loading

AyushExel commented Sep 5, 2022

DN6 commented Sep 5, 2022 •

edited

Loading

AyushExel commented Sep 5, 2022

DN6 commented Sep 5, 2022

AyushExel commented Sep 5, 2022

DN6 commented Sep 6, 2022

DN6 commented Sep 6, 2022

DN6 commented Sep 7, 2022

AyushExel commented Sep 7, 2022

glenn-jocher commented Sep 7, 2022

glenn-jocher commented Sep 7, 2022

glenn-jocher commented Sep 7, 2022 •

edited

Loading

glenn-jocher commented Sep 7, 2022

glenn-jocher commented Sep 7, 2022

DN6 commented Sep 7, 2022 •

edited by glenn-jocher

Loading

glenn-jocher commented Sep 7, 2022

Comet Logging and Visualization Integration #9232

Comet Logging and Visualization Integration #9232

Conversation

DN6 commented Aug 31, 2022 • edited by UltralyticsAssistant Loading

🛠️ PR Summary

🌟 Summary

📊 Key Changes

🎯 Purpose & Impact

DN6 commented Aug 31, 2022 • edited Loading

AyushExel commented Sep 1, 2022

DN6 commented Sep 1, 2022

AyushExel commented Sep 1, 2022

DN6 commented Sep 1, 2022 • edited Loading

AyushExel commented Sep 5, 2022 • edited Loading

AyushExel commented Sep 5, 2022

DN6 commented Sep 5, 2022 • edited Loading

AyushExel commented Sep 5, 2022

DN6 commented Sep 5, 2022

AyushExel commented Sep 5, 2022

DN6 commented Sep 6, 2022

DN6 commented Sep 6, 2022

DN6 commented Sep 7, 2022

AyushExel commented Sep 7, 2022

glenn-jocher commented Sep 7, 2022

glenn-jocher commented Sep 7, 2022

glenn-jocher commented Sep 7, 2022 • edited Loading

glenn-jocher commented Sep 7, 2022

glenn-jocher commented Sep 7, 2022

DN6 commented Sep 7, 2022 • edited by glenn-jocher Loading

glenn-jocher commented Sep 7, 2022

DN6 commented Aug 31, 2022 •

edited by UltralyticsAssistant

Loading

DN6 commented Aug 31, 2022 •

edited

Loading

DN6 commented Sep 1, 2022 •

edited

Loading

AyushExel commented Sep 5, 2022 •

edited

Loading

DN6 commented Sep 5, 2022 •

edited

Loading

glenn-jocher commented Sep 7, 2022 •

edited

Loading

DN6 commented Sep 7, 2022 •

edited by glenn-jocher

Loading