[FEAT] Support GGUF format #2215
Conversation
zhengy001
requested review from
merrymercy,
Ying1123,
zhyncs,
hnyls2002,
ispobock and
ByronHsu
as code owners
|
lm_head.weight is directly used in many places, however, vllm changes it to be |
|
Thanks for the contributions. Can you fix the CI errors? |
zhengy001
force-pushed
the
zyang_dev
branch
2 times, most recently
from
5c616a5 to
2cffa70
Compare
How to trigger the CI? |
Pass lm_head to LogitsProcessor and check the weight inside |
|
@zhengy001 CI won't be triggered for you automatically because you are a first-time contributor. You can send a random typo fix PR and I can merge that for you so your future commits can trigger CI automatically. |
|
@zhengy001 Can you fix the CI errors? |
@merrymercy Sure, working on it. |
|
#2269 adds you as a new contributor so your future commits will trigger CI automatically |
@merrymercy :) |
| # With tie_word_embeddings, we can skip lm_head.weight | ||
| # The weight might appear unnecessarily in the files if the model is | ||
| # processed with quantization, LoRA, fine-tuning, etc. | ||
| if self.config.tie_word_embeddings and "lm_head.weight" in name: |
There was a problem hiding this comment.
There won't be "lm_head.weight" if self.config.tie_word_embeddings is True
| outputs = engine.generate(prompt, sampling_params)["text"] | ||
| engine.shutdown() | ||
|
|
||
| self.assertEqual(outputs, " it. I have a lot of work") |
There was a problem hiding this comment.
Compared the result with vllm's. Pls suggest if there is a better way.
This reverts commit 883c955.
Motivation
#1616
Modifications
Support GGUF format
Checklist