[Model] Added GLM-4 series hf format model support vllm==0.6.4

[sixsixcoder]

Overview

This update adds GLM-4 series text model support vllm==0.6.4, which is different from GLM-4v #9242
This code is written according to transformers==4.46.0

FIX #5306

Usage

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

max_model_len, tp_size = 8192, 1
model_name = "THUDM/glm-4-9b-chat-hf"
prompt = [{"role": "user", "content": "Hello, my name is"}]

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
llm = LLM(
    model=model_name,
    tensor_parallel_size=tp_size,
    max_model_len=max_model_len,
    trust_remote_code=True,
    enforce_eager=True,
)
stop_token_ids = [151329, 151336, 151338]
sampling_params = SamplingParams(temperature=0.95, max_tokens=1024, stop_token_ids=stop_token_ids)

inputs = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)
outputs = llm.generate(prompts=inputs, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[Isotr0py]

Some initial comments.

Since the HF format GLM is highly similar to Llama, I prefer to inherit from existing Llama implementation to reduce code complexity. :)

[sixsixcoder]

Thank you for your reply. I have updated the code according to your suggestion. Can it be merged?

[Isotr0py]

Comparing with the llama implementation, I think we can remove more duplicated codes to simplify the model implementation by inheriting from llama.

[Isotr0py]

Hey @sixsixcoder, I just made a refactor on your GLM implementation to directly inherit from LlamaForCausalLM just like Phi-3, so that we can significantly reduce duplicated codes and support features like PP and LoRA without extra PRs. PTAL!

(Please correct me if there's any issues about model implementation, because you are model vendors and more familiar with GLM than me) :)

[sixsixcoder]

Hey @sixsixcoder, I just made a refactor on your GLM implementation to directly inherit from LlamaForCausalLM just like Phi-3, so that we can significantly reduce duplicated codes and support features like PP and LoRA without extra PRs. PTAL!

(Please correct me if there's any issues about model implementation, because you are model vendors and more familiar with GLM than me) :)

Yes, you are right. Thank you for helping to reorganize the code. I hope it can be merged as soon as possible. This work will enable vllm to support GLM-4 and GLM-Edge series text models.

[Isotr0py]

@sixsixcoder Can you merge from main branch? So that we can get some failing CI fixed.

[DarkLight1337]

Looks good, have you tested this model?

[Isotr0py]

Yes, it can work with both tp=2 and pp=2.

[sixsixcoder]

@sixsixcoder Can you merge from main branch? So that we can get some failing CI fixed.

have merged from main branch, but the CI is still failing. What should I do?

[DarkLight1337]

I think the basic models test failure is from this PR.

[Isotr0py]

Oh, just noticed this model is introduced in transformers >= 4.46.0, have added test skip for it.