Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Model] Add support for the multi-modal Llama 3.2 model #8811

Merged
merged 82 commits into from
Sep 25, 2024
Merged
Changes from 1 commit
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
566d57f
add llamav tokeninizer and redirect loader to it
heheda12345 Aug 30, 2024
218145a
start to load shape
heheda12345 Sep 2, 2024
1c57f26
copy original model
heheda12345 Sep 2, 2024
5233e2d
add LlamaVLConfig
heheda12345 Sep 2, 2024
72b9a8a
can load weight, attention is ignored
heheda12345 Sep 2, 2024
2dd36f5
skip profile run by hardcode, can start model execution
heheda12345 Sep 2, 2024
ba9507d
Merge branch 'main' of github.com:vllm-project/vllm
heheda12345 Sep 2, 2024
affa9ba
can run text tokenizer now
heheda12345 Sep 3, 2024
f633de5
finish image preprocessor
heheda12345 Sep 3, 2024
de8bbad
can run vision encoder now
heheda12345 Sep 4, 2024
30239ad
run prefill self attention
heheda12345 Sep 5, 2024
6972cbf
run prefill crossattention
heheda12345 Sep 6, 2024
4e1344b
can generate the first token :)
heheda12345 Sep 7, 2024
f3d869d
can perform offline e2e run without decode crossattn, but wrong answer
heheda12345 Sep 7, 2024
6f26a3b
pass mm data in encoder-decoder
heheda12345 Sep 8, 2024
fa0912e
prefill result matches now. Model is speaking human words.
heheda12345 Sep 11, 2024
46634ff
generate correct result for single image
heheda12345 Sep 12, 2024
6b73f4d
can support arbitary number of image, need better mask for image_cnt<>1
heheda12345 Sep 12, 2024
fb10a70
temp save for profile run
heheda12345 Sep 12, 2024
718f879
can run tp, but wrong answer
heheda12345 Sep 13, 2024
2644349
can run tp for small model with correct result
heheda12345 Sep 13, 2024
ec4cb9c
tp for vision encoder
heheda12345 Sep 14, 2024
fc01266
update image preprocessor
heheda12345 Sep 15, 2024
3e1d249
support text-only input
heheda12345 Sep 15, 2024
c5ba3cf
Merge tag 'v0.6.1.post2' into llamavl
heheda12345 Sep 15, 2024
cac19d5
enable profile run
heheda12345 Sep 16, 2024
7e5eadd
copy mllama from transformer
heheda12345 Sep 17, 2024
7e3fb1e
can init model from vllm
heheda12345 Sep 17, 2024
49b05d6
weight loader
heheda12345 Sep 17, 2024
2e66a5d
run image encoder now
heheda12345 Sep 18, 2024
9770d84
Add API Server Support
simon-mo Sep 18, 2024
c9d612b
run single image reqeusts correctly
heheda12345 Sep 19, 2024
2f54ae3
single image match huggingface result
heheda12345 Sep 19, 2024
9e2d4ea
Merge remote-tracking branch 'origin/meta-ckpt-early-api-server' into…
heheda12345 Sep 19, 2024
8f3989e
small fix
heheda12345 Sep 19, 2024
01621a5
remove old code
heheda12345 Sep 19, 2024
65a470b
hardcode some config to read huggingface's config.json without modify…
heheda12345 Sep 19, 2024
2146716
move prompt to encoder prompt
heheda12345 Sep 19, 2024
062534b
hardcode to match tokenizer result
heheda12345 Sep 19, 2024
23f04b4
update test script
heheda12345 Sep 20, 2024
4ed4e6e
update test script
heheda12345 Sep 20, 2024
c140258
support text-only input
heheda12345 Sep 21, 2024
f662fdd
fix bug in text only prompt
heheda12345 Sep 21, 2024
6cf166a
add unit test
heheda12345 Sep 21, 2024
b7124e5
add complex tests, but cannot run single-gpu and multi-gpu at the sam…
heheda12345 Sep 21, 2024
e69f127
seperate encoder/decoder dummy input, support max_image=1
heheda12345 Sep 21, 2024
e0e297c
add mllamaconfig to override some params, simplying the model code (WIP)
heheda12345 Sep 22, 2024
f6732cf
upd
heheda12345 Sep 22, 2024
228b66b
code cleanup
heheda12345 Sep 22, 2024
f30319c
remove image processing from input processor
heheda12345 Sep 22, 2024
471e79f
fix precision issue of RMSNorm
heheda12345 Sep 22, 2024
2a0cb7e
only keep usefull vision encoder layer
heheda12345 Sep 22, 2024
f4a7e1e
Merge remote-tracking branch 'public/main' into llamavl
heheda12345 Sep 22, 2024
efbd9b8
merge main
heheda12345 Sep 22, 2024
a596997
format code
heheda12345 Sep 23, 2024
70b6bb3
try formater again
heheda12345 Sep 23, 2024
31000d0
try formater again
heheda12345 Sep 23, 2024
5be8a65
try formater again again again
heheda12345 Sep 23, 2024
8505a8f
try formater again again again again
heheda12345 Sep 23, 2024
a32c3ab
update example
heheda12345 Sep 23, 2024
10d1736
fix bug in openai api -> chat template
heheda12345 Sep 23, 2024
0aa61b0
change model based on new hf
heheda12345 Sep 23, 2024
b993988
make formater happy
heheda12345 Sep 23, 2024
9065770
update model name in example
heheda12345 Sep 23, 2024
bc34aa4
remove mllama chat template, use HF's instead
heheda12345 Sep 23, 2024
a25e383
[Bugfix] Include encoder_prompt_tokens in num_prompt_tokensin UsageInfo
CatherineSue Sep 23, 2024
9b931bf
Merge pull request #6 from vllm-project/chang/num_prompt_tokens
heheda12345 Sep 24, 2024
1eefdc7
update config based on HF update
heheda12345 Sep 24, 2024
ccebf14
Merge branch 'main' of github.com:vllm-project/vllm
heheda12345 Sep 25, 2024
d7750d3
update doc and hf model id
heheda12345 Sep 25, 2024
1ebd6dc
update hf model id again
heheda12345 Sep 25, 2024
3b6fb2b
Merge branch 'main' of github.com:vllm-project/vllm
heheda12345 Sep 25, 2024
c857735
fix format problem
heheda12345 Sep 25, 2024
e4bf803
Apply suggestions from code review
heheda12345 Sep 25, 2024
4d7fe0a
Update vllm/worker/enc_dec_model_runner.py
heheda12345 Sep 25, 2024
4cdc6b5
Update vllm/worker/worker.py
heheda12345 Sep 25, 2024
a6ad79f
Update vllm/worker/worker.py
heheda12345 Sep 25, 2024
8364093
upgrade huggingface
heheda12345 Sep 25, 2024
a12c8d3
Update vllm/transformers_utils/configs/__init__.py
heheda12345 Sep 25, 2024
4065047
update code based on code review
heheda12345 Sep 25, 2024
293f07f
add note
ywang96 Sep 25, 2024
3db294b
format
ywang96 Sep 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
try formater again
  • Loading branch information
heheda12345 committed Sep 23, 2024

Verified

This commit was signed with the committer’s verified signature.
nadouani Nabil Adouani
commit 31000d0d189dd9a0d8e77d25c965b44152905560
10 changes: 4 additions & 6 deletions vllm/transformers_utils/configs/mllama.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
from transformers.models.mllama.configuration_mllama import (MllamaConfig as
MllamaConfigHf)
from transformers.models.mllama.configuration_mllama import (MllamaTextConfig
as
MllamaTextConfigHf
)
from transformers.models.mllama.configuration_mllama import (
MllamaConfig as MllamaConfigHf)
from transformers.models.mllama.configuration_mllama import (
MllamaTextConfig as MllamaTextConfigHf)


class MllamaTextConfig(MllamaTextConfigHf):