Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kebab7 #360

Draft
wants to merge 59 commits into
base: main
Choose a base branch
from
Draft

Kebab7 #360

wants to merge 59 commits into from

Conversation

dirkgr
Copy link
Member

@dirkgr dirkgr commented Nov 3, 2023 •
edited
Loading

This is the kebab config, a smaller version of the dirk config. Differences from dirk:

  • untied weights
  • weight decay on everything
  • adjusted mlp_hidden_size so we come out at 7B parameters including embeddings

Don't merge this until @AkshitaB's config is merged!

AkshitaB and others added 30 commits October 30, 2023 17:04
@AkshitaB
config for no weight tying and weight decay everything
0cba715
@AkshitaB
Merge branch 'main' into kempner-no-weight-tying
9fced7c
@AkshitaB
script to run on kempner
eb7e75b
@AkshitaB
fix data paths in config
7ba7b77
@AkshitaB
update config
68c4029
@AkshitaB
Merge branch 'main' into kempner-no-weight-tying
f4fbf0a
@AkshitaB
checkpointing
4f16792
@AkshitaB
change sharding strategy
f3dda03
@AkshitaB
change group name
33a0338
@dirkgr
Adds an S3 config for the same thing
b35e787
@dirkgr
Merge branch 'kempner-no-weight-tying' of https://github.com/allenai/LLM
99df529 
 into kempner-no-weight-tying
@dirkgr
Fix names according to updates
863bab1
@dirkgr
Correct way of setting the checkpoints path
3f14ce9
@dirkgr
More adaptation, making the two configs the same
ed5f359
@dirkgr
Initial kebap config
07244a5
@dirkgr
More sensible run name
9801816
@dirkgr
Switch to Dolma 1.5
2f2ff0b
@dirkgr
Move value to make the diff nicer
05c357a
@dirkgr
Turn off affine for LN
0261a48
@dirkgr
Enable zloss
5111a39
@dirkgr
Formatting
b7cd4af
@dirkgr
Same FSDP config as Llama
7dc254a
@dirkgr
Non-S3 config for kebap
41d6318
@dirkgr
Kempner run script for kebap7
5a1e54b
@dirkgr
Spelling
b3d207b
@dirkgr
Canceled-check-interval
cd3ad9c
@dirkgr
More workers, more batch size
4a5da9a
@dirkgr
Runs at 1800tps
9d7e351
@dirkgr
Kebab script for LUMI
be90178
@dirkgr
Some more settings for MI250s
b5d9967
@dirkgr
Offline datasets
7e17e91
@dirkgr
Wrap less
8b65329
@dirkgr
This makes things too slow.
e2c2cb9
@dirkgr
Merge remote-tracking branch 'origin/main' into kebab7
a5b5fef 
# Conflicts:
#	olmo/config.py
#	olmo/model.py
@dirkgr
Fastest config we found so far
c625e9e
@dirkgr
Put the time limit back
8af1d74
@dirkgr
Merge remote-tracking branch 'origin/main' into kebab7
d93218a
@dirkgr
Can't load 7B models otherwise
34a2208
@dirkgr
Fix loading certain types of state dicts
739d65a
@dirkgr
Load only one local process at a time
a17b387
@dirkgr
Load optimizer state one at a time
8948270
@dirkgr
Revert "Load optimizer state one at a time"
9527ba3 
This reverts commit 8948270.
@dirkgr
Try one more collect
1830601
@dirkgr
Try a much simpler version of loading
c7916ac
@dirkgr
Revert "Try a much simpler version of loading"
84bbd92 
This reverts commit c7916ac.
@dirkgr
Revert "Try one more collect"
4dde969 
This reverts commit 1830601.
@dirkgr
Revert "Revert "Load optimizer state one at a time""
efe2292 
This reverts commit 9527ba3.
@dirkgr
Revert "Revert "Revert "Load optimizer state one at a time"""
7395c73 
This reverts commit efe2292.
@dirkgr
Revert "Load only one local process at a time"
18f9a6d 
This reverts commit a17b387.
@dirkgr
Optimizer state is broadcasted from rank0
a91c1e1
@dirkgr
Collect some more
f7deae1
@dirkgr
New approach to loading one at a time
fd5ea2a
@dirkgr
Free some more memory sooner
3ebc188
@dirkgr
Merge remote-tracking branch 'origin/main' into kebab7
ee447ff
@dirkgr
Turn off adaptive gradient clipping
b298775
@dirkgr
Some tweaks to the script
4e4766a
@dirkgr
FSDP overlap issue half solved
706ab6c
@dirkgr
8 HW queues seems to make things sloer
c952e9d
@dirkgr
Try torch 2.0 to see if the init thing matters
caea9a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dirkgr @AkshitaB