Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

autoregressive LM and v0.1 release #1

Merged
merged 54 commits into from
Nov 9, 2024
Merged

autoregressive LM and v0.1 release #1

merged 54 commits into from
Nov 9, 2024

Conversation

maxrousseau
Copy link
Owner

everything needed to do causal LM

  • data preparation with datapipelines
  • simple model building blocks
  • composer trainer with sane defaults (lr schedule and gradient clippping)
  • a simple caching system
  • logging with wandb

@maxrousseau
scaffolding, next implement GPT and make package more mature
b0ea14d
@maxrousseau
WIP GPT pythia implementation, early stages
e6c924c
@maxrousseau
iter
5df6f5a
@maxrousseau
iter
6037a05
@maxrousseau
iter
2cc26c9
@maxrousseau
iter
339df42
@maxrousseau
cleanup project a little
a2fbeec
@maxrousseau
forward pass ok, next safetensor and inference eval
407006b
@maxrousseau
pythia weight transfer OK, next TEST inference
f70457b
@maxrousseau
pythia weight transfer OK, next TEST inference
c67eee4
@maxrousseau
rope debugging WIP
91649df
@maxrousseau
fixed RoPE, now pythia tested and working, WIP cleanup
5beff99
@maxrousseau
cleaned up for now, next is tinystories datapipe, training, then KV-c…
d0bd922 
…ache generation
@maxrousseau
iter, WIP tinystories datapipe
6fa46d6
@maxrousseau
test tinystories datapipe
5d086c0
@maxrousseau
datapipe OK for tinystories, next step: train pythia on TS
0f9f9c7
@maxrousseau
training seems to be working, next step full tinystories finetune on GPU
046a680
@maxrousseau
WIP kvcache
57eb263
@maxrousseau
WIP kvcache
065c6d1
@maxrousseau
forgot to add the tests
f0b9d8e
@maxrousseau
moved media
2089ec0
@maxrousseau
moved media
3ac4b00
@maxrousseau
iter
1dd7ae9
@maxrousseau
WIP kv_cache
f937038
@maxrousseau
WIP kv_cache again
5ec5c08
@maxrousseau
kv cache working, next-step is to write test against HF implementation
ee0e0c8
@maxrousseau
found the bug I think, need to implement the UPPER RIGHT causal mask …
8e2559c 
…when caching KVs
@maxrousseau
kvcache implementation is working! -fixed the causal mask
6026370
@maxrousseau
cleanup up the test function a bit
70a7531
@maxrousseau
scuffed out greedy decoding
7f5e190
@maxrousseau
added stop when repeat_ngram
e4003cb
@maxrousseau
iter, training runs work on test, next implement logging and evaluation
58baf7f
@maxrousseau
iter
163b457
@maxrousseau
getting closer to v0.1, logging and wandb is fine next DEBUG=1 and GP…
87ddff3 
…U(s)
@maxrousseau
readme deets
7c39029
@maxrousseau
readme deets
7cbb627
@maxrousseau
iter
ffeb09d
@maxrousseau
some fuckery with uv and torch
3fd69da
@maxrousseau
iter
388e877
@maxrousseau
GPU needs to be debugged, some tensors are allocated on CPU!
fa99cfe
@maxrousseau
fix neoxrope devices
dd899be
@maxrousseau
training on GPU is working
2de297a
@maxrousseau
DEBUG=1 single batch implemented
1624cd7
@maxrousseau
iter
6b796ca
@maxrousseau
iter
70cf05e
@maxrousseau
tinystories full run
8a481ed
@maxrousseau
iter readme
719bccd
@maxrousseau
removed data config, integrated into single config
4553a41
@maxrousseau
readme improvements
81ba73b
@maxrousseau
readme improvements
988660d
@maxrousseau
readme improvements
91f9c27
@maxrousseau
readme improvements
3b37742
@maxrousseau
checkpointing OK, docprompt OK, ready for alpha
7552e5e
@maxrousseau
readme iter
fb1d58b
@maxrousseau maxrousseau merged commit 7759adb into main Nov 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@maxrousseau