[Recipes] Bunch of refactoring #511
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/511
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 0b4f481 with merge base 4f73f75 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for torchtune-preview ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
| opt_state_dict=checkpoint_dict[utils.OPT_KEY] | ||
| if self._resume_from_checkpoint | ||
| else None, | ||
| opt_state_dict=( |
There was a problem hiding this comment.
I don't think our linters that run in CI are consistent with VSCode autoformatters, hence changes like these sneaking in. Shall we look into this? cc @ebsmothers @NicolasHug
There was a problem hiding this comment.
VSCode autoformatter or pre-commit hooks? If the former I think it's low-pri tbh
| if ( | ||
| cfg.full_bf16 | ||
| self._training_precision == torch.bfloat16 |
There was a problem hiding this comment.
Note we also have this utility now
There was a problem hiding this comment.
I don't think I can use that util as-is since it will bail out for CPU devices, but I want this recipe to run on CPU for current CI.
rohan-varma
changed the title
Context
dtypeare the same, fp16 is disabled across recipes, bf16 is checked, etcChangelog
dtypewhich now maps to the "full" low precision as mentioned in [RFC] Configuring low precision training in torchtune #504.log.info(memory_stats_log(...)).Test plan
Follow-ups