Option to disable CodeCarbon

[nebrelbug]

Feature request

• Allow disabling of CodeCarbon reporting, even when it's installed

Motivation

I'm running code using the Trainer class across several nodes in an HPC environment using SLURM. It seems to automatically use the CodeCarbon callback, but unfortunately doesn't work correctly in my multi-node environment and floods my log files with hundreds of distracting messages (mentioned earlier in mlco2/codecarbon#252).

Your contribution

I'd be happy to contribute and/or open a PR, given some guidance about where to start.

[amyeroberts]

Hi @nebrelbug, thanks for raising this issue!

Sounds like a good idea to me. @muellerzr will know more about it than me and can advise.

[muellerzr]

That sounds fine to me as well @nebrelbug.

We have a few options here:

1. Globally disable codecarbon via an environmental variable (similar to DISABLE_WANDB, or something along those lines)
2. Have it as a TrainingArgument (don't quite like that solution)
3. Or: don't add the codecarbon callback if we know we are in a multi-node scenario.

I think I'd prefer 3 if possible, otherwise 1 :) WDYT?

[nebrelbug]

@muellerzr I don't think that CodeCarbon is necessarily broken for everyone using multiple nodes, so my personal preference would be option 1.

Would it be alright if I opened a PR to disable the callback when DISABLE_CODECARBON is set?

[muellerzr]

@nebrelbug actually looking at this more, does using --report_to "none" give the desired result?

[muellerzr]

If so, we should expand the doc on the report_to param in TrainingArguments to mention codecarbon

[nebrelbug]

@muellerzr it does, actually! Setting report_to to either "none" or "wandb" results in CodeCarbon not running.

[muellerzr]

Thanks for verifying! Lets go ahead and add it to the report_to doc string @nebrelbug

[nebrelbug]

@muellerzr sweet! I actually think the doc string for report_to may be complete (though a little hard to find). Instead, I opened up a PR at #26155 to update the Callback doc page. Let me know if that sounds good!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.