Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Deepspeed] add support for bf16 mode #14569

Merged
merged 34 commits into from
Mar 12, 2022
Merged

[Deepspeed] add support for bf16 mode #14569

merged 34 commits into from
Mar 12, 2022

Commits on Nov 30, 2021

  1. [WIP] add support for bf16 mode

    @stas00
    stas00 committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    e449eaf View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2021

  1. Merge remote-tracking branch 'origin/master' into ds-bf16

    @stas00
    stas00 committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    ac337ed View commit details
    Browse the repository at this point in the history

  2. prep for bf16

    @stas00
    stas00 committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    758ed4c View commit details
    Browse the repository at this point in the history

  3. prep for bf16

    @stas00
    stas00 committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    013e44d View commit details
    Browse the repository at this point in the history

  4. fix; zero2/bf16 is ok

    @stas00
    stas00 committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    4cba76b View commit details
    Browse the repository at this point in the history

  5. check bf16 is available

    @stas00
    stas00 committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    85b16d1 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2021

  1. Merge remote-tracking branch 'origin/master' into ds-bf16

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    4d3694c View commit details
    Browse the repository at this point in the history

  2. test fixes

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    e8b69a6 View commit details
    Browse the repository at this point in the history

  3. enable zero3_bf16

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    087ba85 View commit details
    Browse the repository at this point in the history

  4. config files

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    a9c1721 View commit details
    Browse the repository at this point in the history

  5. docs

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    967f383 View commit details
    Browse the repository at this point in the history

  6. split stage_dtype; merge back to non-dtype-specific config file

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    c9e16e6 View commit details
    Browse the repository at this point in the history

  7. fix doc

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    3dec8fd View commit details
    Browse the repository at this point in the history

  8. cleanup

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    ac138b8 View commit details
    Browse the repository at this point in the history

  9. cleanup

    @stas00
    stas00 committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    8b516e3 View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2021

  1. Merge remote-tracking branch 'origin/master' into ds-bf16

    @stas00
    stas00 committed Dec 10, 2021
    Configuration menu
    Copy the full SHA
    a403c31 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2022

  1. merge/port to mdx

    @stas00
    stas00 committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    dd9b8b9 View commit details
    Browse the repository at this point in the history

  2. bfloat16 => bf16 to match the PR changes

    @stas00
    stas00 committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    d48f68e View commit details
    Browse the repository at this point in the history

  3. s/zero_gather_fp16_weights_on_model_save/zero_gather_16bit_weights_on… 

    …_model_save/; s/save_fp16_model/save_16bit_model/
    @stas00
    stas00 committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    0fca057 View commit details
    Browse the repository at this point in the history

  4. test fixes/skipping

    @stas00
    stas00 committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    41176e4 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2022

  1. Merge remote-tracking branch 'origin/master' into ds-bf16

    @stas00
    stas00 committed Jan 28, 2022
    Configuration menu
    Copy the full SHA
    f0d53c8 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into ds-bf16

    @stas00
    stas00 committed Jan 28, 2022
    Configuration menu
    Copy the full SHA
    5a63855 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2022

  1. move

    @stas00
    stas00 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    9c20180 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/master' into ds-bf16

    @stas00
    stas00 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    1f93706 View commit details
    Browse the repository at this point in the history

  3. fix

    @stas00
    stas00 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    9f50e90 View commit details
    Browse the repository at this point in the history

  4. Update docs/source/main_classes/deepspeed.mdx 

    Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
    @stas00 @sgugger
    stas00 and sgugger committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    4ea8cab View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2022

  1. backticks

    @stas00
    stas00 committed Feb 12, 2022
    Configuration menu
    Copy the full SHA
    b8df847 View commit details
    Browse the repository at this point in the history

  2. cleanup

    @stas00
    stas00 committed Feb 12, 2022
    Configuration menu
    Copy the full SHA
    7062bb2 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'ds-bf16' of github.com:stas00/transformers into ds-bf16

    @stas00
    stas00 committed Feb 12, 2022
    Configuration menu
    Copy the full SHA
    31ce3fd View commit details
    Browse the repository at this point in the history

  4. cleanup

    @stas00
    stas00 committed Feb 12, 2022
    Configuration menu
    Copy the full SHA
    fcc40d3 View commit details
    Browse the repository at this point in the history

  5. cleanup

    @stas00
    stas00 committed Feb 12, 2022
    Configuration menu
    Copy the full SHA
    8529f87 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2022

  1. Merge remote-tracking branch 'origin/master' into ds-bf16

    @stas00
    stas00 committed Mar 12, 2022
    Configuration menu
    Copy the full SHA
    54b6c46 View commit details
    Browse the repository at this point in the history

  2. new version

    @stas00
    stas00 committed Mar 12, 2022
    Configuration menu
    Copy the full SHA
    7cb6455 View commit details
    Browse the repository at this point in the history

  3. add note about grad accum in bf16

    @stas00
    stas00 committed Mar 12, 2022
    Configuration menu
    Copy the full SHA
    75fc6ce View commit details
    Browse the repository at this point in the history