Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kebab7 #360

Draft
wants to merge 59 commits into
base: main
Choose a base branch
from
Draft

Kebab7 #360

wants to merge 59 commits into from

Commits on Oct 30, 2023

  1. config for no weight tying and weight decay everything

    @AkshitaB
    AkshitaB committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    0cba715 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into kempner-no-weight-tying

    @AkshitaB
    AkshitaB committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    9fced7c View commit details
    Browse the repository at this point in the history

  3. script to run on kempner

    @AkshitaB
    AkshitaB committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    eb7e75b View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. fix data paths in config

    @AkshitaB
    AkshitaB committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    7ba7b77 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2023

  1. update config

    @AkshitaB
    AkshitaB committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    68c4029 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2023

  1. Merge branch 'main' into kempner-no-weight-tying

    @AkshitaB
    AkshitaB committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    f4fbf0a View commit details
    Browse the repository at this point in the history

  2. checkpointing

    @AkshitaB
    AkshitaB committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    4f16792 View commit details
    Browse the repository at this point in the history

  3. change sharding strategy

    @AkshitaB
    AkshitaB committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    f3dda03 View commit details
    Browse the repository at this point in the history

  4. change group name

    @AkshitaB
    AkshitaB committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    33a0338 View commit details
    Browse the repository at this point in the history

  5. Adds an S3 config for the same thing

    @dirkgr
    dirkgr committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    b35e787 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'kempner-no-weight-tying' of https://github.com/allenai/LLM 

     into kempner-no-weight-tying
    @dirkgr
    dirkgr committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    99df529 View commit details
    Browse the repository at this point in the history

  7. Fix names according to updates

    @dirkgr
    dirkgr committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    863bab1 View commit details
    Browse the repository at this point in the history

  8. Correct way of setting the checkpoints path

    @dirkgr
    dirkgr committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    3f14ce9 View commit details
    Browse the repository at this point in the history

  9. More adaptation, making the two configs the same

    @dirkgr
    dirkgr committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    ed5f359 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. Initial kebap config

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    07244a5 View commit details
    Browse the repository at this point in the history

  2. More sensible run name

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    9801816 View commit details
    Browse the repository at this point in the history

  3. Switch to Dolma 1.5

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    2f2ff0b View commit details
    Browse the repository at this point in the history

  4. Move value to make the diff nicer

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    05c357a View commit details
    Browse the repository at this point in the history

  5. Turn off affine for LN

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    0261a48 View commit details
    Browse the repository at this point in the history

  6. Enable zloss

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    5111a39 View commit details
    Browse the repository at this point in the history

  7. Formatting

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    b7cd4af View commit details
    Browse the repository at this point in the history

  8. Same FSDP config as Llama

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    7dc254a View commit details
    Browse the repository at this point in the history

  9. Non-S3 config for kebap

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    41d6318 View commit details
    Browse the repository at this point in the history

  10. Kempner run script for kebap7

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    5a1e54b View commit details
    Browse the repository at this point in the history

  11. Spelling

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    b3d207b View commit details
    Browse the repository at this point in the history

  12. Canceled-check-interval

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    cd3ad9c View commit details
    Browse the repository at this point in the history

  13. More workers, more batch size

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    4a5da9a View commit details
    Browse the repository at this point in the history

  14. Runs at 1800tps

    @dirkgr
    dirkgr committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    9d7e351 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. Kebab script for LUMI

    @dirkgr
    dirkgr committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    be90178 View commit details
    Browse the repository at this point in the history

  2. Some more settings for MI250s

    @dirkgr
    dirkgr committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    b5d9967 View commit details
    Browse the repository at this point in the history

  3. Offline datasets

    @dirkgr
    dirkgr committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    7e17e91 View commit details
    Browse the repository at this point in the history

  4. Wrap less

    @dirkgr
    dirkgr committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    8b65329 View commit details
    Browse the repository at this point in the history

  5. This makes things too slow.

    @dirkgr
    dirkgr committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    e2c2cb9 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/main' into kebab7 

    # Conflicts:
#	olmo/config.py
#	olmo/model.py
    @dirkgr
    dirkgr committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    a5b5fef View commit details
    Browse the repository at this point in the history

  7. Fastest config we found so far

    @dirkgr
    dirkgr committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    c625e9e View commit details
    Browse the repository at this point in the history

  8. Put the time limit back

    @dirkgr
    dirkgr committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    8af1d74 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. Merge remote-tracking branch 'origin/main' into kebab7

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    d93218a View commit details
    Browse the repository at this point in the history

  2. Can't load 7B models otherwise

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    34a2208 View commit details
    Browse the repository at this point in the history

  3. Fix loading certain types of state dicts

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    739d65a View commit details
    Browse the repository at this point in the history

  4. Load only one local process at a time

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    a17b387 View commit details
    Browse the repository at this point in the history

  5. Load optimizer state one at a time

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    8948270 View commit details
    Browse the repository at this point in the history

  6. Revert "Load optimizer state one at a time" 

    This reverts commit 8948270.
    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    9527ba3 View commit details
    Browse the repository at this point in the history

  7. Try one more collect

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    1830601 View commit details
    Browse the repository at this point in the history

  8. Try a much simpler version of loading

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    c7916ac View commit details
    Browse the repository at this point in the history

  9. Revert "Try a much simpler version of loading" 

    This reverts commit c7916ac.
    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    84bbd92 View commit details
    Browse the repository at this point in the history

  10. Revert "Try one more collect" 

    This reverts commit 1830601.
    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    4dde969 View commit details
    Browse the repository at this point in the history

  11. Revert "Revert "Load optimizer state one at a time"" 

    This reverts commit 9527ba3.
    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    efe2292 View commit details
    Browse the repository at this point in the history

  12. Revert "Revert "Revert "Load optimizer state one at a time""" 

    This reverts commit efe2292.
    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    7395c73 View commit details
    Browse the repository at this point in the history

  13. Revert "Load only one local process at a time" 

    This reverts commit a17b387.
    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    18f9a6d View commit details
    Browse the repository at this point in the history

  14. Optimizer state is broadcasted from rank0

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    a91c1e1 View commit details
    Browse the repository at this point in the history

  15. Collect some more

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    f7deae1 View commit details
    Browse the repository at this point in the history

  16. New approach to loading one at a time

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    fd5ea2a View commit details
    Browse the repository at this point in the history

  17. Free some more memory sooner

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    3ebc188 View commit details
    Browse the repository at this point in the history

  18. Merge remote-tracking branch 'origin/main' into kebab7

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    ee447ff View commit details
    Browse the repository at this point in the history

  19. Turn off adaptive gradient clipping

    @dirkgr
    dirkgr committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    b298775 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. Some tweaks to the script

    @dirkgr
    dirkgr committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    4e4766a View commit details
    Browse the repository at this point in the history

  2. FSDP overlap issue half solved

    @dirkgr
    dirkgr committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    706ab6c View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2023

  1. 8 HW queues seems to make things sloer

    @dirkgr
    dirkgr committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    c952e9d View commit details
    Browse the repository at this point in the history

  2. Try torch 2.0 to see if the init thing matters

    @dirkgr
    dirkgr committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    caea9a2 View commit details
    Browse the repository at this point in the history