Fix same family model issue and add test #9

xuechendi · 2024-11-27T22:15:37Z

No description provided.

github-actions · 2024-11-27T22:15:50Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

Signed-off-by: youkaichao <youkaichao@gmail.com>

…roject#10682) Signed-off-by: Ubuntu <zixuanzhang@bytedance.com>

…t#10701) Signed-off-by: spacewander <spacewanderlzx@gmail.com>

…ject#10724) Signed-off-by: rickyx <rickyx@anyscale.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Richard Liu <ricliu@google.com>

…project#10561) Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…-project#10742) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

…ject#10733) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Isotr0py <2037008807@qq.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

…-project#10771) Signed-off-by: Isotr0py <2037008807@qq.com>

Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com>

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

…0769) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…ject#10763) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…0805) Signed-off-by: youkaichao <youkaichao@gmail.com>

…oject#10804) Signed-off-by: Roger Wang <ywang@roblox.com> Co-authored-by: Chen Zhang <zhangch99@outlook.com> Co-authored-by: Isotr0py <2037008807@qq.com>

…t#10502) This PR provides initial support for single-node disaggregated prefill in 1P1D scenario. Signed-off-by: KuntaiDu <kuntai@uchicago.edu> Co-authored-by: ApostaC <yihua98@uchicago.edu> Co-authored-by: YaoJiayi <120040070@link.cuhk.edu.cn>

Signed-off-by: Isotr0py <2037008807@qq.com>

vllm-project#10809)

…or (vllm-project#10834) Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

… prompts (vllm-project#10753)

…oject#10842) Signed-off-by: Ubuntu <zixuanzhang@bytedance.com>

…10844) Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Signed-off-by: youkaichao <youkaichao@gmail.com>

…oject#10838) Signed-off-by: youkaichao <youkaichao@gmail.com>

Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…it as default (vllm-project#10785) Signed-off-by: Aaron Pham <contact@aarnphm.xyz> Signed-off-by: mgoin <michael@neuralmagic.com> Co-authored-by: mgoin <michael@neuralmagic.com>

…vllm-project#10850) Co-authored-by: Yang Zheng(SW)(Alex) <you@example.com>

…lm-project#10640) Signed-off-by: Roger Wang <ywang@roblox.com> Co-authored-by: Michael Goin <michael@neuralmagic.com> Co-authored-by: Roger Wang <ywang@roblox.com>

Signed-off-by: yan ma <yan.ma@intel.com>

Signed-off-by: mgoin <michael@neuralmagic.com>

…ject#10853) Signed-off-by: Tomer Asida <tomera@ai21.com>

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>

…0557) Signed-off-by: Aaron Pham <contact@aarnphm.xyz> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Co-authored-by: Aaron Pham <contact@aarnphm.xyz>

…t#10876) Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

…oject#10854) Signed-off-by: jianzheng <57654625+o2363286@users.noreply.github.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

xuechendi force-pushed the multi_models_rebase branch from 17af66c to 3b76699 Compare November 27, 2024 23:36

jikunshang force-pushed the rebase_mm branch from 1e11d26 to d3a20b6 Compare November 28, 2024 02:01

youkaichao and others added 27 commits November 27, 2024 19:54

[misc] upgrade filelock version (vllm-project#10731)

cb4e1c3

Signed-off-by: youkaichao <youkaichao@gmail.com>

[Model] support bitsandbytes quantization with minicpm3 model (vllm-p…

70dc14f

…roject#10682) Signed-off-by: Ubuntu <zixuanzhang@bytedance.com>

[Doc] Update model in arch_overview.rst to match comment (vllm-projec…

278be67

…t#10701) Signed-off-by: spacewander <spacewanderlzx@gmail.com>

[Bug][CLI] Allow users to disable prefix caching explicitly (vllm-pro…

d9b4b3f

…ject#10724) Signed-off-by: rickyx <rickyx@anyscale.com>

[V1] Do not allocate beyond the max_model_len (vllm-project#10730)

a79b122

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Kernel] Update vllm-flash-attn version (vllm-project#10736)

9a8bff0

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[TPU] Update requirements-tpu (vllm-project#10726)

3ed5e73

Signed-off-by: Richard Liu <ricliu@google.com>

[Model] Added GLM-4 series hf format model support vllm==0.6.4 (vllm-…

5fc5ce0

…project#10561) Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

[Kernel] Update vllm-flash-attn version to reduce CPU overheads (vllm…

8c1e77f

…-project#10742) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[V1] Optimize the CPU overheads in FlashAttention custom op (vllm-pro…

98f47f2

…ject#10733) Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

[Model] Add Internlm2 LoRA support (vllm-project#5064)

c83919c

Signed-off-by: Isotr0py <2037008807@qq.com>

[Model] Clean up MiniCPMV (vllm-project#10751)

fa6ecb9

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Misc] typo find in sampling_metadata.py (vllm-project#10740)

c82b432

[Bugfix] Fix Idefics3 bug (vllm-project#10778)

3132aac

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

[platform] Add verify_quantization in platform. (vllm-project#10757)

661175b

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

[Bugfix] Fix OpenVino/Neuron driver_worker init (vllm-project#10779)

40bc242

Signed-off-by: NickLucche <nlucches@redhat.com> Signed-off-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>

[Model] Refactor Molmo weights loading to use AutoWeightsLoader (vllm…

16ee07f

…-project#10771) Signed-off-by: Isotr0py <2037008807@qq.com>

[Interleaved ATTN] Support for Mistral-8B (vllm-project#10591)

e7cfc4e

Signed-off-by: youkaichao <youkaichao@gmail.com> Co-authored-by: youkaichao <youkaichao@gmail.com>

[doc] format fix (vllm-project#10789)

7e4bbda

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

[Model] Replace embedding models with pooling adapter (vllm-project#1…

1337071

…0769) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Misc] Improve type annotations for support_torch_compile (vllm-pro…

f877a7d

…ject#10763) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Misc] Rename embedding classes to pooling (vllm-project#10801)

d2f058e

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[doc] add warning about comparing hf and vllm outputs (vllm-project#1…

169a0ff

…0805) Signed-off-by: youkaichao <youkaichao@gmail.com>

[Misc] Adding MMMU-Pro vision dataset to serving benchmark (vllm-pr…

c11f172

…oject#10804) Signed-off-by: Roger Wang <ywang@roblox.com> Co-authored-by: Chen Zhang <zhangch99@outlook.com> Co-authored-by: Isotr0py <2037008807@qq.com>

[Model] Add BNB support to Llava and Pixtral-HF (vllm-project#10795)

b18c9bb

Signed-off-by: Isotr0py <2037008807@qq.com>

[core] Avoid metrics log noise when idle - include speculative decodi… (

b795477

vllm-project#10809)

Fix openvino on GPU (vllm-project#10793)

9b14d97

xuechendi force-pushed the multi_models_rebase branch from 168af3c to 3133992 Compare December 2, 2024 22:44

[Model] Add TP and BNB quantization support to LlavaMultiModalProject…

4c05edb

…or (vllm-project#10834) Signed-off-by: Isotr0py <2037008807@qq.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

xuechendi force-pushed the multi_models_rebase branch from 3133992 to 0915219 Compare December 2, 2024 23:31

mgoin and others added 21 commits December 3, 2024 02:26

[Bugfix] Prevent benchmark_throughput.py from using duplicated random…

4433195

… prompts (vllm-project#10753)

[Model] support bitsandbytes quantization with minicpm model (vllm-pr…

d746268

…oject#10842) Signed-off-by: Ubuntu <zixuanzhang@bytedance.com>

[Bugfix] Fix QKVParallelLinearWithShardedLora bias bug (vllm-project#…

a4cf256

…10844) Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

[core][distributed] add pynccl broadcast (vllm-project#10843)

21fe7b4

Signed-off-by: youkaichao <youkaichao@gmail.com>

[torch.compile] remove compilation_context and simplify code (vllm-pr…

dc5ce86

…oject#10838) Signed-off-by: youkaichao <youkaichao@gmail.com>

[Doc] Add github links for source code references (vllm-project#10672)

ef51831

Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Misc] Remove deprecated names (vllm-project#10817)

3257d44

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Core][Performance] Add XGrammar support for guided decoding and set …

9323a31

…it as default (vllm-project#10785) Signed-off-by: Aaron Pham <contact@aarnphm.xyz> Signed-off-by: mgoin <michael@neuralmagic.com> Co-authored-by: mgoin <michael@neuralmagic.com>

[Speculative Decoding] Move indices to device before filtering output (…

f6084f6

…vllm-project#10850) Co-authored-by: Yang Zheng(SW)(Alex) <you@example.com>

[V1] VLM - Run the mm_mapper preprocessor in the frontend process (vl…

3bc94ca

…lm-project#10640) Signed-off-by: Roger Wang <ywang@roblox.com> Co-authored-by: Michael Goin <michael@neuralmagic.com> Co-authored-by: Roger Wang <ywang@roblox.com>

[MISC][XPU] quick fix for XPU CI (vllm-project#10859)

2f2cdc7

Signed-off-by: yan ma <yan.ma@intel.com>

[Bugfix] Only require XGrammar on x86 (vllm-project#10865)

7090c27

Signed-off-by: mgoin <michael@neuralmagic.com>

[Frontend] correctly record prefill and decode time metrics (vllm-pro…

7c32b68

…ject#10853) Signed-off-by: Tomer Asida <tomera@ai21.com>

[Build][Bugfix] Using the correct type hint (vllm-project#10866)

a061fe6

Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>

[Benchmark] Benchmark structured output with datasets (vllm-project#1…

381ac93

…0557) Signed-off-by: Aaron Pham <contact@aarnphm.xyz> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Co-authored-by: Aaron Pham <contact@aarnphm.xyz>

[CI/Build] Replace mean with torch.all in test_pynccl.py (vllm-projec…

d2bd88b

…t#10876) Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

Drop ROCm load format check (vllm-project#10767)

b5b647b

Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>

[ci/build] Change queue name for Release jobs (vllm-project#10875)

fa2dea6

[ci/build] Job to build and push release image (vllm-project#10877)

c9ca4fc

[bugfix] fixed parameter “n” when set parameter “bestof” > 1 (vllm-pr…

8db957e

…oject#10854) Signed-off-by: jianzheng <57654625+o2363286@users.noreply.github.com>

[ci/build] Update vLLM postmerge ECR repo (vllm-project#10887)

c92acb9

xuechendi force-pushed the multi_models_rebase branch from 0915219 to 655b610 Compare December 4, 2024 17:16

jikunshang and others added 4 commits December 4, 2024 20:10

copy origin file to mm_ files

41685cb

support multi models

883fac0

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> Signed-off-by: Chendi Xue <chendi.xue@intel.com> Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

add model arg to original client

655b610

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>

fix GPU memory allocation for multi_models

e4ad1c5

Signed-off-by: Chendi Xue <chendi.xue@intel.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix same family model issue and add test #9

Fix same family model issue and add test #9

xuechendi commented Nov 27, 2024

github-actions bot commented Nov 27, 2024

Fix same family model issue and add test #9

Are you sure you want to change the base?

Fix same family model issue and add test #9

Conversation

xuechendi commented Nov 27, 2024

github-actions bot commented Nov 27, 2024