bf16+pipeline parallelism #1801
Conversation
bf16 checkpoint save/load
tjruwase
requested review from
samyam,
conglongli,
awan-10,
cli99,
eltonzheng,
minjiaz and
RezaYazdaniAminabadi
as code owners
|
@stas00, FYI |
tjruwase
force-pushed
the
olruwase/bf16-updates
branch
from
ed26ef4
to
27e5b95
Compare
…d into olruwase/bf16-updates
…d into olruwase/bf16-updates
…d into olruwase/bf16-updates
| @@ -981,6 +969,10 @@ def _configure_distributed_model(self, model): | |||
| hasattr(param, | |||
| 'ds_id') for param in self.module.parameters()): | |||
| self.__check_params(self.module, torch.bfloat16) | |||
| if self.zero_optimization_stage() == 0 and not self.pipeline_parallelism: | |||
| raise NotImplementedError( | |||
| "When not running ZeRO, BF16 training support is only supported for Pipeline parallelism" | |||
There was a problem hiding this comment.
Hello, I wonder why BF16 is only supported for Pipeline parallelism or ZeRO 1 to ZeRO3, since there is not such limit in prior version.
There was a problem hiding this comment.
@kisseternity, apologies for the confusion here. This is a new bf16+Pipeline parallelism code path that was written in the last minute for BLOOM model training. The existing restrictions in combining with ZeRO are temporary. We plan to harmonize these combinations and eliminate the confusions.
There was a problem hiding this comment.
@kisseternity, apologies for the confusion here. This is a new
bf16+Pipeline parallelismcode path that was written in the last minute for BLOOM model training. The existing restrictions in combining with ZeRO are temporary. We plan to harmonize these combinations and eliminate the confusions.
Thanks for replying. In that case, I think bf16 can be used without the bf16 optimizer or ZeRO as before.
There was a problem hiding this comment.
Yes, these changes do not affect the previous support for bf16+ZeRO.
bf16_optimizer implementing optimizer state sharding (a.k.a., zero stage 1)
Integration with pipeline parallelism