Skip to content
This repository has been archived by the owner on Oct 19, 2024. It is now read-only.

[EXAMPLE] Port Codegen model into alpa #774

Merged
merged 54 commits into from
Feb 7, 2023
Merged

[EXAMPLE] Port Codegen model into alpa #774

merged 54 commits into from
Feb 7, 2023

Conversation

BabyChouSr
Copy link
Contributor

@BabyChouSr BabyChouSr commented Nov 13, 2022 •
edited
Loading

Context: Trying to migrate codegen model into alpa

The main code in codegen_model.py comes from modeling_flax_gptj.py

@BabyChouSr
codegen model init
30a0d8c
@BabyChouSr
add codegen prefix to model
fe8cbb3
@BabyChouSr
add codegen into wrapper
2c5b964
@BabyChouSr
remove autoregressive
058477f
@BabyChouSr
add codegen to textgen
f63d2e8
@BabyChouSr
redo codegen configs inside model
8ad39c4
@BabyChouSr
rename params in codegen
d9a6fdf
@BabyChouSr
remove extra n
310f41f
@BabyChouSr
replace max target positions by max seq len
96e7ea1
@BabyChouSr
refactor loading params
cc5eb96
@BabyChouSr
replace decoder embed dims with hidden size
5ac2d58
@BabyChouSr
replace decoder layers with num_hidden_layers
b586029
@BabyChouSr
replace decoder attention heads with n_head
e79216b
@BabyChouSr
added some config params
9d353c1
@BabyChouSr
add deterministic for dropout
c0b66ea
@BabyChouSr
fix params loading
83aca90
@BabyChouSr
fix more config lines
2d1801c
@BabyChouSr
Merge remote-tracking branch 'origin/pr-codegen-llm' into pr-codegen-llm
d494212
@BabyChouSr
add out proj and dont use bias
aefd87e
@BabyChouSr
fix embeddings
ded6f15
@BabyChouSr
replace position ids with token_type
d95007a
@BabyChouSr
delete redundancies
6389f98
@BabyChouSr
fix code blocks to mirror gptj
eb9d589
@BabyChouSr
fix codegenattention
6a360ea
@BabyChouSr
fix param pass through
5a98c60
@BabyChouSr
fix mismatched shapes and transformer module
76858ea
@BabyChouSr
add position ids to params and fix precision
973ee8b
@BabyChouSr
Merge branch 'pr-codegen-llm' of https://github.com/BabyChouSr/alpa i…
d4d4cbb 
…nto pr-codegen-llm
@BabyChouSr
fix vocab size
de998c9
@BabyChouSr
remove layer and fix weights
c04291b
@merrymercy merrymercy assigned merrymercy and unassigned merrymercy Dec 23, 2022
@BabyChouSr BabyChouSr marked this pull request as ready for review February 7, 2023 16:55
@merrymercy
Copy link
Member

merrymercy commented Feb 7, 2023 •
edited
Loading

LGTM. I applied some small tweaks. Thanks for the contribution!

@merrymercy merrymercy merged commit bec4fae into alpa-projects:main Feb 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@merrymercy merrymercy merrymercy left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@BabyChouSr @merrymercy