don't use no_sync when deepspeed doesn't support it for certain zero stages #35157
Conversation
winglian
force-pushed
the
deepspeed-no-sync-zero
branch
from
7ef3da1 to
63acec4
Compare
|
might have to broaden deepspeed for all zero cases? https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/engine.py#L2208-L2209 |
src/transformers/trainer.py
Outdated
| context = ( | ||
| functools.partial(self.accelerator.no_sync, model=model) | ||
| if i != len(batch_samples) - 1 | ||
| if i != len(batch_samples) - 1 and not disable_deepspeed_no_sync |
There was a problem hiding this comment.
| if i != len(batch_samples) - 1 and not disable_deepspeed_no_sync | |
| if i != len(batch_samples) - 1 and not self.accelerator.distributed_type == DistributedType.DEEPSPEED |
There was a problem hiding this comment.
may i ask which version of transformers support this fix-up? mine is 4.46.0. same problem with deepspeed 0.16
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Thanks! We've also confirmed this fixes up all the fails users reported wrt deepspeed. cc @ArthurZucker for final post wing doing quality ;)
|
can you just run |
…stages (#35157) * don't use no_sync when deepspeed doesn't support it for certain zero stages * chore: lint * fix no_sync context for deepspeed across all zero types * chore: lint
…stages (huggingface#35157) * don't use no_sync when deepspeed doesn't support it for certain zero stages * chore: lint * fix no_sync context for deepspeed across all zero types * chore: lint
|
Strangely, this issue still exists on |
|
This is on 4.48 ! |
What does this PR do?
Deepspeed 0.16 has assertions preventing the use of no_sync with zero 2/3. see https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/engine.py#L1986-L2004
it seems people are reporting this here microsoft/DeepSpeed#6793, and I'm assuming that everyone is using accelerate/transformers as downgrading to deepspeed 0.15.4 makes it "work" for them.
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@muellerzr
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.