[Performance] Enable chunked prefill and prefix caching together #7753

comaniac · 2024-08-21T19:21:26Z

Reference PRs: #6144, #6819
Make @sighingnow and @Juelianqvq as co-authors of this PR.

This PR supports prefix caching and chunked prefill to be enabled together. Different from the reference PRs, this PR simplifies the logic of dealing with partial blocks (thanks to @rkooo567 for the suggestion). Here is the execution flow:

In scheduler, when determining the new tokens to be scheduled and both chunked prefill and prefix caching are enabled.
1. If all uncomputed tokens can be scheduled (i.e., the last chunk of the prompt), then schedule them all.
2. Otherwise, we always schedule the number of tokens that is divisible by the block size. For example, if the remaining budget is 133 tokens and the block size is 16, we will only schedule (133//16)*16=112 tokens. Although this approach wastes some token budget, it makes the following process straightforward.
In prepare input, if all scheduled tokens are cached, we only compute the last block. Note that:
1. We cannot skip all blocks at this moment because model runner doesn't support this case. Currently when block manager determines prefix cache blocks, it will also skip the last block due to the same reason (e.g., https://github.com/vllm-project/vllm/blob/main/vllm/core/block/prefix_caching_block.py#L556). This can be improved in the future if we move prefix caching to scheduler so that this case won't happen anymore.
2. Since we guarantee the scheduled tokens are divisible by block size, we don't need to consider partial blocks in prepare input.

A test case for functional correctness is also added.

Throughput benchmarking results:

Model: neuralmagic/Meta-Llama-3-8B-Instruct-FP8
GPU: 1xL4
Number of requests: 600
Average prompt length: 637 (shared prefix ~180, cache hit rate ~20%)
Max output length: 200
Block manager v1
Chunked prefill size 2048

Branch	ChunkedPrefill	PrefixCaching	Elapsed Time (s)	Throughput (tok/s)
main	x	v	154.37	3631.2
main	v	x	173.84	3215.1
PR	x	v	155.88	3596.2
PR	v	x	174.18	3298.8
PR	v	v	142.81	3929.7

cc @rkooo567

github-actions · 2024-08-21T19:21:38Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

Comment /ready on the PR
Add ready label to the PR
Enable auto-merge.

🚀

rkooo567 · 2024-08-21T19:28:59Z

result seems very good!!

sighingnow · 2024-08-21T20:47:38Z

Hi @comaniac @rkooo567 I would like you folks to notice my last commit on #6144 (a043643).

Without it, this PR is still incorrect, and the error can be reproduced with even a single request:

request 1: length 120
chunked prefill enabled
prefix caching enabled
max_num_batched_tokens = 64, max_num_seqs = 64

You will find that with this PR, at the first round, tokens[0:64] is prefilled, at the second round, tokens[96:119] is prefilled, and the tokens between 64 and 96 are skipped.

This is because the num_computed_blocks is incorrectly updated as the whole block table for prompt tokens, rather than tokens that are prefilled at the first round.

comaniac · 2024-08-21T20:53:55Z

IIUC, this PR already guarantees every sequence will have at least one block to compute even it fully hits the cache, so it shouldn't trigger the issue you mentioned? If I missed anything, can you modify the unit test added in this PR so that the problem can be exposed and tested?

sighingnow · 2024-08-21T20:56:31Z

IIUC, this PR already guarantees every sequence will have at least one block to compute even it fully hits the cache, so it shouldn't trigger the issue you mentioned?

It is not about fully matched. In the case commented above, there are only 1 request, and the prefill are spited to [0:64] and [64:120], and the second part is treated as prefix matched as the computed_block_nums are updated to [0,1,2,3,4,5,6,7] after the first chunk prefill.

sighingnow · 2024-08-21T20:58:45Z

IIUC, this PR already guarantees every sequence will have at least one block to compute even it fully hits the cache, so it shouldn't trigger the issue you mentioned? If I missed anything, can you modify the unit test added in this PR so that the problem can be exposed and tested?

The test case in this PR didn't fail just because the max_num_batched_tokens (14) is smaller than the block size (16). Try larger value like 64.

comaniac · 2024-08-21T21:01:38Z

IIUC, this PR already guarantees every sequence will have at least one block to compute even it fully hits the cache, so it shouldn't trigger the issue you mentioned? If I missed anything, can you modify the unit test added in this PR so that the problem can be exposed and tested?

The test case in this PR didn't fail just because the max_num_batched_tokens (14) is smaller than the block size (16). Try larger value like 64.

The size 14 is used to test invalid size. The actual size being tested in this case is 16. Meanwhile, I tried all 16, 32 and 64 but none of them failed.

sighingnow · 2024-08-21T21:06:10Z

IIUC, this PR already guarantees every sequence will have at least one block to compute even it fully hits the cache, so it shouldn't trigger the issue you mentioned? If I missed anything, can you modify the unit test added in this PR so that the problem can be exposed and tested?

The test case in this PR didn't fail just because the max_num_batched_tokens (14) is smaller than the block size (16). Try larger value like 64.

The size 14 is used to test invalid size. The actual size being tested in this case is 16. Meanwhile, I tried all 16, 32 and 64 but none of them failed.

With max_num_batched_tokens=64, you need sequence length at least to 64 + 2 * block_size to reproduce the problem, 41 is not enough.

max_num_batched_tokens=16/32 cannot reproduce the issue, too, as the second block are guaranteed to be recomputed in this PR.

comaniac · 2024-08-21T21:21:37Z

Ok I could reproduce the issue you pointed out. It actually only happens in block manager v1 as block manager v2 doesn't use this mechanism to mark computed blocks. This may also explain the too good speedup I got. I'll apply your fix in this PR and try to make the test cover this case.

comaniac · 2024-08-21T23:37:12Z

@sighingnow I applied your commit with some modifications. The test is also changed so that it will fail without fixing the issue in block manager v1. PTAL.

sighingnow · 2024-08-22T00:16:23Z

@sighingnow I applied your commit with some modifications. The test is also changed so that it will fail without fixing the issue in block manager v1. PTAL.

Thanks! LGTM.

rkooo567

Looks good. One question is should we just make scheduler handle prefix caching + chunked prefill correctly and make logics in model_runner simplified?

vllm/core/block_manager_v1.py

rkooo567 · 2024-08-22T00:32:25Z

vllm/core/scheduler.py

+                    raise ValueError("When enabling chunked prefill and "
+                                     "prefix caching, max_num_batched_tokens "
+                                     "(chunk size) must be dividable by "
+                                     "block size, but got "


can you also print chunk size and block size along with budget.token_budget % block_size?

It now looks like

ValueError: When enabling chunked prefill and prefix caching, max_num_batched_tokens (chunk size) must be dividable by block size, but got chunk_size (30) % block_size (16) = 14

vllm/core/scheduler.py

vllm/worker/model_runner.py

sighingnow · 2024-08-22T06:14:14Z

@sighingnow I applied your commit with some modifications. The test is also changed so that it will fail without fixing the issue in block manager v1. PTAL.

Will the fix for v2 block manager be addressed by this PR as well? The behavior of v2-block-manager looks quite strange and I'm wondering if #7619 is related.

comaniac · 2024-08-22T06:24:04Z

@sighingnow I applied your commit with some modifications. The test is also changed so that it will fail without fixing the issue in block manager v1. PTAL.

Will the fix for v2 block manager be addressed by this PR as well? The behavior of v2-block-manager looks quite strange and I'm wondering if #7619 is related.

I have a fix in my local but it would be a separate PR

JaheimLee · 2024-08-22T08:50:00Z

Is it for flash-attn backend only or for all backends?

comaniac · 2024-08-22T15:35:33Z

Is it for flash-attn backend only or for all backends?

I've tested flash-attn and FlashInfer so at least these 2 backends work. Need to test xformers later.

Juelianqvq · 2024-08-23T02:00:24Z

I've tested flash-attn and FlashInfer so at least these 2 backends work. Need to test xformers later.

@comaniac https://github.com/vllm-project/vllm/blob/main/vllm/attention/backends/flashinfer.py#L360 Really supported here?

comaniac · 2024-08-23T02:09:42Z

I've tested flash-attn and FlashInfer so at least these 2 backends work. Need to test xformers later.

@comaniac https://github.com/vllm-project/vllm/blob/main/vllm/attention/backends/flashinfer.py#L360 Really supported here?

Yeah I noticed that too so not fully sure what's going on. Will find some time tomorrow for it.

comaniac · 2024-08-23T19:38:54Z

Updates:

More tests are added.
Chunk prefill does only support flash attention backend for now. My local test passed because it didn't schedule prefill and decode in the same batch. However, there shouldn't be a blocker for FlashInfer to support chunked prefill, so we should add this support in a follow-up PR.

sighingnow · 2024-08-24T12:53:38Z

Updates:

More tests are added.

Chunk prefill does only support flash attention backend for now. My local test passed because it didn't schedule prefill and decode in the same batch. However, there shouldn't be a blocker for FlashInfer to support chunked prefill, so we should add this support in a follow-up PR.

May I know more why you choose to recompute the whole block if it is fully matched? Only recompute the last token is enough and requires no changes in scheduler, and it would be a bit more efficient.

comaniac · 2024-08-24T17:20:44Z

You're right it would be a bit more efficient to compute only the last token. Meanwhile I found that it might not be that hard to deal with prefix matching in scheduler so that this case would never happen in model runner. I'll give it a try

Co-authored-by: Tao He <sighingnow@gmail.com> Co-authored-by: Juelianqvq <Juelianqvq@noreply.github.com>

comaniac · 2024-08-26T21:09:51Z

@sighingnow changed to re-compute only the last token. PTAL.

@rkooo567 I've tried to move prefix caching to scheduler and it's actually easy for default scheduler. For chunked prefill, we have to refactor the scheduler (e.g., .schedule(), ._schedule_prefill(), .get_new_tokens()) and block manager (e.g., .can_allocate()). Since we have to be careful with this refactor and it can be decoupled from this PR, I'll put it in a follow-up PR tracked by #7883

rkooo567

Generally looks good. I'd like to actually also add a warning if the block size is big and prefix caching + CP is enabled (because it can waste a lot of tokens). Maybe if block_size >32, we can print a warning?

rkooo567 · 2024-08-28T06:27:35Z

tests/core/test_block_manager.py

@@ -595,3 +595,43 @@ def test_sliding_window_multi_seq():

    # assert all blocks are free now
    assert block_manager.get_num_free_gpu_blocks() == num_gpu_blocks
+
+
+def test_mark_blocks_as_computed_with_prefix_cache_and_chunked_prefill():


do we have corresponding test in v2?

We don't need to test v2 because v2 automatically mark touched blocks as computed.

rkooo567 · 2024-08-28T06:32:32Z

vllm/core/scheduler.py

+                # to avoid partial block matching.
+                block_size = self.cache_config.block_size
+                reminder = budget.token_budget % block_size
+                if reminder != 0:


Btw, should we raise this exception at the engine start time instead and just add assert here?

I feel we could just raise here for now because this constraint should be able to be removed once we refactor the schedule to consider prefix caching.

comaniac · 2024-08-28T07:36:26Z

Generally looks good. I'd like to actually also add a warning if the block size is big and prefix caching + CP is enabled (because it can waste a lot of tokens). Maybe if block_size >32, we can print a warning?

Sure I'll add the warning in a follow-up PR.

Juelianqvq · 2024-08-28T12:56:22Z

Since this PR has been merged, both #6144 and #6819 can be closed, and are you willing to add me and @sighingnow as the co-authors? @comaniac

comaniac · 2024-08-28T13:21:58Z

Ah I intended to do that. Actually I put you two as co-authors in one commit of this PR and I thought it should work when the PR is merged but somehow it didn't...let me try to figure out how to fix that. Also cc @simon-mo

…m-project#7753)

sighingnow · 2024-08-29T02:59:22Z

To whom it may concern: after this PR there are still occasional crashes when prefix caching and chunked prefill are enabled at the same time on Nvidia GPUs (inside the flash_attn_varlen_func function in the prefix-enabled attention branch). I investigated the kernel input and find nothing wrong and cannot reproduce it when run the kernel standalone with the pickle saved inputs. I think there are still overflow bugs inside vllm-flash-attention, set the block_size to 256 could fix the issue and the crash disappeared under high pressure.

Co-authored-by: Tao He <sighingnow@gmail.com> Co-authored-by: Juelianqvq <Juelianqvq@noreply.github.com>

ashgold · 2024-09-03T23:13:08Z

To whom it may concern: after this PR there are still occasional crashes when prefix caching and chunked prefill are enabled at the same time on Nvidia GPUs (inside the flash_attn_varlen_func function in the prefix-enabled attention branch). I investigated the kernel input and find nothing wrong and cannot reproduce it when run the kernel standalone with the pickle saved outputs. I think there are still overflow bugs inside vllm-flash-attention, set the block_size to 256 could fix the issue and the crash disappeared under high pressure.

This looks like a serious bug that needs to be fixed before it can go to production. Thanks for sharing the workaround solution as well.

…m-project#7753)

commit a1d8742 Author: Simon Mo <simon.mo@hey.com> Date: Mon Sep 9 23:21:00 2024 -0700 Add NVIDIA Meetup slides, announce AMD meetup, and add contact info (vllm-project#8319) commit 6cd5e5b Author: Dipika Sikka <dipikasikka1@gmail.com> Date: Mon Sep 9 23:02:52 2024 -0400 [Misc] Fused MoE Marlin support for GPTQ (vllm-project#8217) commit c7cb5c3 Author: Kyle Sayers <kylesayrs@gmail.com> Date: Mon Sep 9 16:27:26 2024 -0400 [Misc] GPTQ Activation Ordering (vllm-project#8135) commit f9b4a2d Author: Vladislav Kruglikov <vladislavkruglikov@outlook.com> Date: Mon Sep 9 21:20:46 2024 +0300 [Bugfix] Correct adapter usage for cohere and jamba (vllm-project#8292) commit 58fcc85 Author: Adam Lugowski <alugowski@gmail.com> Date: Mon Sep 9 11:16:37 2024 -0700 [Frontend] Add progress reporting to run_batch.py (vllm-project#8060) Co-authored-by: Adam Lugowski <adam.lugowski@parasail.io> commit 08287ef Author: Kyle Mistele <kyle@mistele.com> Date: Mon Sep 9 09:45:11 2024 -0500 [Bugfix] Streamed tool calls now more strictly follow OpenAI's format; ensures Vercel AI SDK compatibility (vllm-project#8272) commit 4ef41b8 Author: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com> Date: Sun Sep 8 00:01:51 2024 -0400 [Bugfix] Fix async postprocessor in case of preemption (vllm-project#8267) commit cfe712b Author: Joe Runde <Joseph.Runde@ibm.com> Date: Sat Sep 7 14:03:16 2024 -0600 [CI/Build] Use python 3.12 in cuda image (vllm-project#8133) Signed-off-by: Joe Runde <Joseph.Runde@ibm.com> commit b962ee1 Author: sumitd2 <91451282+sumitd2@users.noreply.github.com> Date: Sat Sep 7 23:48:40 2024 +0530 ppc64le: Dockerfile fixed, and a script for buildkite (vllm-project#8026) commit 36bf815 Author: Isotr0py <2037008807@qq.com> Date: Sun Sep 8 01:45:44 2024 +0800 [Model][VLM] Decouple weight loading logic for `Paligemma` (vllm-project#8269) commit e807125 Author: Isotr0py <2037008807@qq.com> Date: Sat Sep 7 16:38:23 2024 +0800 [Model][VLM] Support multi-images inputs for InternVL2 models (vllm-project#8201) commit 9f68e00 Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Sat Sep 7 16:02:39 2024 +0800 [Bugfix] Fix broken OpenAI tensorizer test (vllm-project#8258) commit ce2702a Author: youkaichao <youkaichao@gmail.com> Date: Fri Sep 6 22:40:46 2024 -0700 [tpu][misc] fix typo (vllm-project#8260) commit 795b662 Author: Wei-Sheng Chin <wschin@outlook.com> Date: Fri Sep 6 20:18:16 2024 -0700 Enable Random Prefix Caching in Serving Profiling Tool (benchmark_serving.py) (vllm-project#8241) commit 2f707fc Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Sat Sep 7 10:57:24 2024 +0800 [Model] Multi-input support for LLaVA (vllm-project#8238) commit 41e95c5 Author: Kyle Mistele <kyle@mistele.com> Date: Fri Sep 6 21:49:01 2024 -0500 [Bugfix] Fix Hermes tool call chat template bug (vllm-project#8256) Co-authored-by: Kyle Mistele <kyle@constellate.ai> commit 12dd715 Author: William Lin <SolitaryThinker@users.noreply.github.com> Date: Fri Sep 6 17:48:48 2024 -0700 [misc] [doc] [frontend] LLM torch profiler support (vllm-project#7943) commit 29f49cd Author: Patrick von Platen <patrick.v.platen@gmail.com> Date: Sat Sep 7 01:02:05 2024 +0200 [Model] Allow loading from original Mistral format (vllm-project#8168) Co-authored-by: Michael Goin <michael@neuralmagic.com> commit 23f3222 Author: Dipika Sikka <dipikasikka1@gmail.com> Date: Fri Sep 6 18:29:03 2024 -0400 [Misc] Remove `SqueezeLLM` (vllm-project#8220) commit 9db52ea Author: rasmith <Randall.Smith@amd.com> Date: Fri Sep 6 17:26:09 2024 -0500 [Kernel] [Triton] Memory optimization for awq_gemm and awq_dequantize, 2x throughput (vllm-project#8248) commit 1447c97 Author: Alexey Kondratiev(AMD) <143633163+alexeykondrat@users.noreply.github.com> Date: Fri Sep 6 14:51:03 2024 -0400 [CI/Build] Increasing timeout for multiproc worker tests (vllm-project#8203) commit de80783 Author: Rui Qiao <161574667+ruisearch42@users.noreply.github.com> Date: Fri Sep 6 09:18:35 2024 -0700 [Misc] Use ray[adag] dependency instead of cuda (vllm-project#7938) commit e5cab71 Author: afeldman-nm <156691304+afeldman-nm@users.noreply.github.com> Date: Fri Sep 6 12:01:14 2024 -0400 [Frontend] Add --logprobs argument to `benchmark_serving.py` (vllm-project#8191) commit baa5467 Author: Nick Hill <nickhill@us.ibm.com> Date: Thu Sep 5 20:39:29 2024 -0700 [BugFix] Fix Granite model configuration (vllm-project#8216) commit db3bf7c Author: Jiaxin Shan <seedjeffwan@gmail.com> Date: Thu Sep 5 18:10:33 2024 -0700 [Core] Support load and unload LoRA in api server (vllm-project#6566) Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> commit 2febcf2 Author: sroy745 <142070531+sroy745@users.noreply.github.com> Date: Thu Sep 5 13:25:29 2024 -0700 [Documentation][Spec Decode] Add documentation about lossless guarantees in Speculative Decoding in vLLM (vllm-project#7962) commit 2ee4528 Author: Michael Goin <michael@neuralmagic.com> Date: Thu Sep 5 11:09:46 2024 -0400 Move verify_marlin_supported to GPTQMarlinLinearMethod (vllm-project#8165) commit 9da25a8 Author: Alex Brooks <alex.brooks@ibm.com> Date: Thu Sep 5 06:48:10 2024 -0600 [MODEL] Qwen Multimodal Support (Qwen-VL / Qwen-VL-Chat) (vllm-project#8029) Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com> Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk> commit 8685ba1 Author: manikandan.tm@zucisystems.com <94887255+Manikandan-Thangaraj-ZS0321@users.noreply.github.com> Date: Thu Sep 5 17:03:37 2024 +0530 Inclusion of InternVLChatModel In PP_SUPPORTED_MODELS(Pipeline Parallelism) (vllm-project#7860) commit 288a938 Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Thu Sep 5 18:51:53 2024 +0800 [Doc] Indicate more information about supported modalities (vllm-project#8181) commit e39ebf5 Author: Elfie Guo <164945471+elfiegg@users.noreply.github.com> Date: Wed Sep 4 22:12:26 2024 -0700 [Core/Bugfix] Add query dtype as per FlashInfer API requirements. (vllm-project#8173) commit ba262c4 Author: Kevin H. Luu <kevin@anyscale.com> Date: Wed Sep 4 20:33:12 2024 -0700 [ci] Mark LoRA test as soft-fail (vllm-project#8160) Signed-off-by: kevin <kevin@anyscale.com> commit 4624d98 Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Wed Sep 4 20:31:48 2024 -0700 [Misc] Clean up RoPE forward_native (vllm-project#8076) commit 1afc931 Author: William Lin <SolitaryThinker@users.noreply.github.com> Date: Wed Sep 4 17:35:36 2024 -0700 [bugfix] >1.43 constraint for openai (vllm-project#8169) Co-authored-by: Michael Goin <michael@neuralmagic.com> commit e01c2be Author: Maureen McElaney <mmcelaney@users.noreply.github.com> Date: Wed Sep 4 19:50:13 2024 -0400 [Doc] [Misc] Create CODE_OF_CONDUCT.md (vllm-project#8161) commit 32e7db2 Author: Simon Mo <simon.mo@hey.com> Date: Wed Sep 4 16:34:27 2024 -0700 Bump version to v0.6.0 (vllm-project#8166) commit 008cf88 Author: Harsha vardhan manoj Bikki <39381063+hbikki@users.noreply.github.com> Date: Wed Sep 4 16:33:43 2024 -0700 [Neuron] Adding support for adding/ overriding neuron configuration a… (vllm-project#8062) Co-authored-by: Harsha Bikki <harbikh@amazon.com> commit 77d9e51 Author: Cody Yu <hao.yu.cody@gmail.com> Date: Wed Sep 4 13:23:22 2024 -0700 [MISC] Replace input token throughput with total token throughput (vllm-project#8164) Co-authored-by: Michael Goin <michael@neuralmagic.com> commit e02ce49 Author: Kyle Mistele <kyle@mistele.com> Date: Wed Sep 4 15:18:13 2024 -0500 [Feature] OpenAI-Compatible Tools API + Streaming for Hermes & Mistral models (vllm-project#5649) Co-authored-by: constellate <constellate@1-ai-appserver-staging.codereach.com> Co-authored-by: Kyle Mistele <kyle@constellate.ai> commit 561d6f8 Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Wed Sep 4 13:05:50 2024 -0700 [CI] Change test input in Gemma LoRA test (vllm-project#8163) commit d1dec64 Author: alexeykondrat <143633163+alexeykondrat@users.noreply.github.com> Date: Wed Sep 4 14:57:54 2024 -0400 [CI/Build][ROCm] Enabling LoRA tests on ROCm (vllm-project#7369) Co-authored-by: Simon Mo <simon.mo@hey.com> commit 2ad2e56 Author: Cody Yu <hao.yu.cody@gmail.com> Date: Wed Sep 4 11:53:25 2024 -0700 [MISC] Consolidate FP8 kv-cache tests (vllm-project#8131) commit d331156 Author: wnma <wnma3mz@gmail.com> Date: Wed Sep 4 18:55:37 2024 +0800 [Bugfix] remove post_layernorm in siglip (vllm-project#8106) commit ccd7207 Author: TimWang <7367474+haitwang-cloud@users.noreply.github.com> Date: Wed Sep 4 14:17:05 2024 +0800 chore: Update check-wheel-size.py to read MAX_SIZE_MB from env (vllm-project#8103) commit 855c262 Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Wed Sep 4 13:22:17 2024 +0800 [Frontend] Multimodal support in offline chat (vllm-project#8098) commit 2be8ec6 Author: Peter Salas <peter@fixie.ai> Date: Tue Sep 3 21:38:21 2024 -0700 [Model] Add Ultravox support for multiple audio chunks (vllm-project#7963) commit e16fa99 Author: Dipika Sikka <dipikasikka1@gmail.com> Date: Tue Sep 3 22:12:41 2024 -0400 [Misc] Update fbgemmfp8 to use `vLLMParameters` (vllm-project#7972) Co-authored-by: Michael Goin <michael@neuralmagic.com> commit 61f4a93 Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Tue Sep 3 18:35:33 2024 -0700 [TPU][Bugfix] Use XLA rank for persistent cache path (vllm-project#8137) commit d4db9f5 Author: Nick Hill <nickhill@us.ibm.com> Date: Tue Sep 3 17:57:41 2024 -0700 [Benchmark] Add `--async-engine` option to benchmark_throughput.py (vllm-project#7964) commit 2188a60 Author: Dipika Sikka <dipikasikka1@gmail.com> Date: Tue Sep 3 17:21:44 2024 -0400 [Misc] Update `GPTQ` to use `vLLMParameters` (vllm-project#7976) commit dc0b606 Author: Simon Mo <simon.mo@hey.com> Date: Tue Sep 3 14:11:42 2024 -0700 [CI] Change PR remainder to avoid at-mentions (vllm-project#8134) commit 0af3abe Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Tue Sep 3 13:29:24 2024 -0700 [TPU][Bugfix] Fix next_token_ids shape (vllm-project#8128) commit f1575dc Author: Kevin H. Luu <kevin@anyscale.com> Date: Tue Sep 3 13:25:09 2024 -0700 [ci] Fix GHA workflow (vllm-project#8129) Signed-off-by: kevin <kevin@anyscale.com> commit c02638e Author: tomeras91 <57313761+tomeras91@users.noreply.github.com> Date: Tue Sep 3 22:37:08 2024 +0300 [CI/Build] make pip install vllm work in macos (for import only) (vllm-project#8118) commit 652c83b Author: Antoni Baum <antoni.baum@protonmail.com> Date: Tue Sep 3 12:28:25 2024 -0700 [Misc] Raise a more informative exception in add/remove_logger (vllm-project#7750) commit 6d646d0 Author: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com> Date: Tue Sep 3 14:50:29 2024 -0400 [Core] Optimize Async + Multi-step (vllm-project#8050) commit 95a178f Author: Kevin H. Luu <kevin@anyscale.com> Date: Tue Sep 3 11:32:27 2024 -0700 [CI] Only PR reviewers/committers can trigger CI on PR (vllm-project#8124) Signed-off-by: kevin <kevin@anyscale.com> commit bd852f2 Author: Cody Yu <hao.yu.cody@gmail.com> Date: Tue Sep 3 10:49:18 2024 -0700 [Performance] Enable chunked prefill and prefix caching together (vllm-project#8120) Co-authored-by: Tao He <sighingnow@gmail.com> Co-authored-by: Juelianqvq <Juelianqvq@noreply.github.com> commit ec26653 Author: Isotr0py <2037008807@qq.com> Date: Tue Sep 3 21:37:52 2024 +0800 [Bugfix][VLM] Add fallback to SDPA for ViT model running on CPU backend (vllm-project#8061) commit 0fbc669 Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Mon Sep 2 20:35:42 2024 -0700 [Bugfix] Fix single output condition in output processor (vllm-project#7881) commit 6e36f4f Author: wang.yuqi <noooop@126.com> Date: Tue Sep 3 05:20:12 2024 +0800 improve chunked prefill performance [Bugfix] Fix vllm-project#7592 vllm 0.5.4 enable_chunked_prefill throughput is slightly lower than 0.5.3~0.5.0. (vllm-project#7874) commit dd2a6a8 Author: Isotr0py <2037008807@qq.com> Date: Mon Sep 2 23:48:56 2024 +0800 [Bugfix] Fix internlm2 tensor parallel inference (vllm-project#8055) commit 4ca65a9 Author: Isotr0py <2037008807@qq.com> Date: Mon Sep 2 20:43:26 2024 +0800 [Core][Bugfix] Accept GGUF model without .gguf extension (vllm-project#8056) commit e2b2aa5 Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Sun Sep 1 23:09:46 2024 -0700 [TPU] Align worker index with node boundary (vllm-project#7932) commit e6a26ed Author: Lily Liu <lilyliupku@gmail.com> Date: Sun Sep 1 21:23:29 2024 -0700 [SpecDecode][Kernel] Flashinfer Rejection Sampling (vllm-project#7244) commit f8d6014 Author: Shawn Tan <shawn@wtf.sg> Date: Sun Sep 1 21:37:18 2024 -0400 [Model] Add Granite model (vllm-project#7436) Co-authored-by: Nick Hill <nickhill@us.ibm.com> commit 5b86b19 Author: Roger Wang <136131678+ywang96@users.noreply.github.com> Date: Sun Sep 1 14:46:57 2024 -0700 [Misc] Optional installation of audio related packages (vllm-project#8063) commit 5231f08 Author: Roger Wang <136131678+ywang96@users.noreply.github.com> Date: Sat Aug 31 16:35:53 2024 -0700 [Frontend][VLM] Add support for multiple multi-modal items (vllm-project#8049) commit 8423aef Author: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com> Date: Sat Aug 31 15:44:03 2024 -0400 [BugFix][Core] Multistep Fix Crash on Request Cancellation (vllm-project#8059) commit 4f5d844 Author: Nicolò Lucchesi <nicolo.lucchesi@gmail.com> Date: Sat Aug 31 09:27:58 2024 +0200 [Bugfix] Fix ModelScope models in v0.5.5 (vllm-project#8037) commit d05f0a9 Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Sat Aug 31 13:26:55 2024 +0800 [Bugfix] Fix import error in Phi-3.5-MoE (vllm-project#8052) commit 622f8ab Author: Pavani Majety <pmajety@nvidia.com> Date: Fri Aug 30 22:18:50 2024 -0700 [Bugfix] bugfix and add model test for flashinfer fp8 kv cache. (vllm-project#8013) commit 1248e85 Author: Wenxiang <8460860+wenxcs@users.noreply.github.com> Date: Sat Aug 31 03:42:57 2024 +0800 [Model] Adding support for MSFT Phi-3.5-MoE (vllm-project#7729) Co-authored-by: Your Name <you@example.com> Co-authored-by: Zeqi Lin <zelin@microsoft.com> Co-authored-by: Zeqi Lin <Zeqi.Lin@microsoft.com> commit 2684efc Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Fri Aug 30 09:01:26 2024 -0700 [TPU][Bugfix] Fix tpu type api (vllm-project#8035) commit 058344f Author: Kaunil Dhruv <dhruv.kaunil@gmail.com> Date: Fri Aug 30 08:21:02 2024 -0700 [Frontend]-config-cli-args (vllm-project#7737) Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Kaunil Dhruv <kaunil_dhruv@intuit.com> commit 98cef6a Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Fri Aug 30 23:20:34 2024 +0800 [Core] Increase default `max_num_batched_tokens` for multimodal models (vllm-project#8028) commit f97be32 Author: Jungho Christopher Cho <wjdgh6655@gmail.com> Date: Sat Aug 31 00:19:27 2024 +0900 [VLM][Model] TP support for ViTs (vllm-project#7186) Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com> Co-authored-by: Roger Wang <ywang@roblox.com> commit afd39a4 Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Fri Aug 30 23:03:28 2024 +0800 [Bugfix] Fix import error in Exaone model (vllm-project#8034) commit 2148441 Author: Richard Liu <39319471+richardsliu@users.noreply.github.com> Date: Fri Aug 30 00:27:40 2024 -0700 [TPU] Support single and multi-host TPUs on GKE (vllm-project#7613) commit dc13e99 Author: Yohan Na <nayohan13@gmail.com> Date: Fri Aug 30 15:34:20 2024 +0900 [MODEL] add Exaone model support (vllm-project#7819) commit 34a0e96 Author: Avshalom Manevich <12231371+avshalomman@users.noreply.github.com> Date: Fri Aug 30 11:11:39 2024 +0700 [Kernel] changing fused moe kernel chunk size default to 32k (vllm-project#7995) commit 80c7b08 Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Thu Aug 29 19:35:29 2024 -0700 [TPU] Async output processing for TPU (vllm-project#8011) commit 428dd14 Author: afeldman-nm <156691304+afeldman-nm@users.noreply.github.com> Date: Thu Aug 29 22:19:08 2024 -0400 [Core] Logprobs support in Multi-step (vllm-project#7652) commit 4abed65 Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Fri Aug 30 08:49:04 2024 +0800 [VLM] Disallow overflowing `max_model_len` for multimodal models (vllm-project#7998) commit 0c785d3 Author: Wei-Sheng Chin <wechi@microsoft.com> Date: Thu Aug 29 16:48:11 2024 -0700 Add more percentiles and latencies (vllm-project#7759) commit 4664cea Author: chenqianfzh <51831990+chenqianfzh@users.noreply.github.com> Date: Thu Aug 29 16:09:08 2024 -0700 support bitsandbytes 8-bit and FP4 quantized models (vllm-project#7445) commit 257afc3 Author: Harsha vardhan manoj Bikki <39381063+hbikki@users.noreply.github.com> Date: Thu Aug 29 13:58:14 2024 -0700 [Neuron] Adding support for context-lenght, token-gen buckets. (vllm-project#7885) Co-authored-by: Harsha Bikki <harbikh@amazon.com> commit 86a677d Author: Dipika Sikka <dipikasikka1@gmail.com> Date: Thu Aug 29 16:46:55 2024 -0400 [misc] update tpu int8 to use new vLLM Parameters (vllm-project#7973) commit d78789a Author: Isotr0py <2037008807@qq.com> Date: Fri Aug 30 03:54:49 2024 +0800 [Bugfix] Fix incorrect vocal embedding shards for GGUF model in tensor parallelism (vllm-project#7954) commit c334b18 Author: kushanam <42385577+kushanam@users.noreply.github.com> Date: Thu Aug 29 12:15:04 2024 -0700 extend cuda graph size for H200 (vllm-project#7894) Co-authored-by: youkaichao <youkaichao@126.com> commit 6b34215 Author: Pavani Majety <pavanimajety@gmail.com> Date: Thu Aug 29 11:53:11 2024 -0700 [Core][Kernels] Enable FP8 KV Cache with Flashinfer backend. + BugFix for kv_cache_dtype=auto (vllm-project#7985) Co-authored-by: Simon Mo <simon.mo@hey.com> Co-authored-by: Cody Yu <hao.yu.cody@gmail.com> commit 3f60f22 Author: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com> Date: Thu Aug 29 14:18:26 2024 -0400 [Core] Combine async postprocessor and multi-step (vllm-project#7921) commit f205c09 Author: Jonas M. Kübler <44084297+jmkuebler@users.noreply.github.com> Date: Thu Aug 29 07:18:13 2024 +0200 [Bugfix] Unify rank computation across regular decoding and speculative decoding (vllm-project#7899) commit ef99a78 Author: youkaichao <youkaichao@gmail.com> Date: Wed Aug 28 21:27:06 2024 -0700 Revert "[Core][Kernels] Use FlashInfer backend for FP8 KV Cache when available." (vllm-project#7982) commit 74d5543 Author: Peter Salas <peter@fixie.ai> Date: Wed Aug 28 20:24:31 2024 -0700 [VLM][Core] Fix exceptions on ragged NestedTensors (vllm-project#7974) commit a7f65c2 Author: youkaichao <youkaichao@gmail.com> Date: Wed Aug 28 17:32:26 2024 -0700 [torch.compile] remove reset (vllm-project#7975) commit 4289cad Author: Nick Hill <nickhill@us.ibm.com> Date: Wed Aug 28 17:22:43 2024 -0700 [Frontend] Minor optimizations to zmq decoupled front-end (vllm-project#7957) Co-authored-by: Robert Shaw <rshaw@neuralmagic> commit af59df0 Author: Michael Goin <michael@neuralmagic.com> Date: Wed Aug 28 19:19:17 2024 -0400 Remove faulty Meta-Llama-3-8B-Instruct-FP8.yaml lm-eval test (vllm-project#7961) commit ce6bf3a Author: youkaichao <youkaichao@gmail.com> Date: Wed Aug 28 16:10:12 2024 -0700 [torch.compile] avoid Dynamo guard evaluation overhead (vllm-project#7898) Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> commit 3cdfe1f Author: bnellnm <49004751+bnellnm@users.noreply.github.com> Date: Wed Aug 28 18:11:49 2024 -0400 [Bugfix] Make torch registration of punica ops optional (vllm-project#7970) commit fdd9daa Author: Mor Zusman <mor.zusmann@gmail.com> Date: Thu Aug 29 01:06:52 2024 +0300 [Kernel/Model] Migrate mamba_ssm and causal_conv1d kernels to vLLM (vllm-project#7651) commit 8c56e57 Author: Stas Bekman <stas00@users.noreply.github.com> Date: Wed Aug 28 13:54:23 2024 -0700 [Doc] fix 404 link (vllm-project#7966) commit eeffde1 Author: Woosuk Kwon <woosuk.kwon@berkeley.edu> Date: Wed Aug 28 13:10:21 2024 -0700 [TPU] Upgrade PyTorch XLA nightly (vllm-project#7967) commit e5697d1 Author: rasmith <Randall.Smith@amd.com> Date: Wed Aug 28 14:37:47 2024 -0500 [Kernel] [Triton] [AMD] Adding Triton implementations awq_dequantize and awq_gemm to support AWQ (vllm-project#7386) commit b98cc28 Author: Pavani Majety <pavanimajety@gmail.com> Date: Wed Aug 28 10:01:22 2024 -0700 [Core][Kernels] Use FlashInfer backend for FP8 KV Cache when available. (vllm-project#7798) Co-authored-by: Simon Mo <simon.mo@hey.com> commit ef9baee Author: Cyrus Leung <tlleungac@connect.ust.hk> Date: Wed Aug 28 23:11:18 2024 +0800 [Bugfix][VLM] Fix incompatibility between vllm-project#7902 and vllm-project#7230 (vllm-project#7948) commit 98c12cf Author: Stas Bekman <stas00@users.noreply.github.com> Date: Wed Aug 28 05:12:32 2024 -0700 [Doc] fix the autoAWQ example (vllm-project#7937) commit f52a43a Author: youkaichao <youkaichao@gmail.com> Date: Wed Aug 28 01:27:07 2024 -0700 [ci][test] fix pp test failure (vllm-project#7945) commit e358053 Author: Cody Yu <hao.yu.cody@gmail.com> Date: Wed Aug 28 00:36:31 2024 -0700 [Performance] Enable chunked prefill and prefix caching together (vllm-project#7753)

hmellor · 2024-09-10T14:14:10Z

If you are using a model with max_model_len > 32K (i.e. Llama 3.1) then chunked prefill is enabled by default. However, this PR leaves the and not self.enable_prefix_caching condition in this automatic enabling of chunked prefill.

This means that a user relying on the automatic enabling of chunked prefill might not notice it becoming disabled when they enable prefix caching.

vllm/vllm/engine/arg_utils.py

Lines 866 to 891 in da1a844

    
           if self.enable_chunked_prefill is None: 
        
               # If not explicitly set, enable chunked prefill by default for 
        
               # long context (> 32K) models. This is to avoid OOM errors in the 
        
               # initial memory profiling phase. 
        
               if use_long_context: 
        
                   is_gpu = device_config.device_type == "cuda" 
        
                   use_sliding_window = (model_config.get_sliding_window() 
        
                                         is not None) 
        
                   use_spec_decode = self.speculative_model is not None 
        
                   has_seqlen_agnostic_layers = ( 
        
                       model_config.contains_seqlen_agnostic_layers( 
        
                           parallel_config)) 
        
                   if (is_gpu and not use_sliding_window and not use_spec_decode 
        
                           and not self.enable_lora 
        
                           and not self.enable_prompt_adapter 
        
                           and not self.enable_prefix_caching 
        
                           and not has_seqlen_agnostic_layers): 
        
                       self.enable_chunked_prefill = True 
        
                       logger.warning( 
        
                           "Chunked prefill is enabled by default for models with " 
        
                           "max_model_len > 32K. Currently, chunked prefill might " 
        
                           "not work with some features or models. If you " 
        
                           "encounter any issues, please disable chunked prefill " 
        
                           "by setting --enable-chunked-prefill=False.") 
        
               if self.enable_chunked_prefill is None: 
        
                   self.enable_chunked_prefill = False

cc @comaniac

comaniac · 2024-09-10T15:33:48Z

Good point. I'll file another PR to fix it.

…m-project#7753)

…m-project#7753) Signed-off-by: Alvant <alvasian@yandex.ru>

…m-project#7753)

comaniac requested review from zhuohan123 and rkooo567 August 21, 2024 19:22

comaniac changed the title ~~Prefix cache chunked prefill~~ [Performance] Enable chunked prefill and prefix caching together Aug 21, 2024

rkooo567 reviewed Aug 22, 2024

View reviewed changes

comaniac and others added 6 commits August 26, 2024 11:25

done

d893717

test

1f16ece

Add co-authors

94315d4

Co-authored-by: Tao He <sighingnow@gmail.com> Co-authored-by: Juelianqvq <Juelianqvq@noreply.github.com>

final

1daa758

fix

79563bf

clean up

f1e9548

comaniac added 2 commits August 26, 2024 11:26

comments and tests

d57951f

computel ast

324fcec

comaniac force-pushed the prefix-cache-chunked-prefill branch from b305e0d to 324fcec Compare August 26, 2024 19:59

comaniac added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 26, 2024

rkooo567 approved these changes Aug 28, 2024

View reviewed changes

comaniac merged commit e358053 into vllm-project:main Aug 28, 2024
54 checks passed

comaniac deleted the prefix-cache-chunked-prefill branch August 28, 2024 07:36

kushanam pushed a commit to kushanam/vllm that referenced this pull request Aug 28, 2024

[Performance] Enable chunked prefill and prefix caching together (vll…

1fcd098

…m-project#7753)

hbikki pushed a commit to hbikki/vllm that referenced this pull request Aug 28, 2024

[Performance] Enable chunked prefill and prefix caching together (vll…

60b3dc0

…m-project#7753)

kushanam pushed a commit to kushanam/vllm that referenced this pull request Aug 28, 2024

[Performance] Enable chunked prefill and prefix caching together (vll…

2497d44

…m-project#7753)

flozi00 mentioned this pull request Sep 3, 2024

[WIP] Multi Step Chunked Prefill - Prefill Steps #8001

Closed

comaniac added a commit to comaniac/vllm that referenced this pull request Sep 3, 2024

Add co-authors of vllm-project#7753

f13313c

Co-authored-by: Tao He <sighingnow@gmail.com> Co-authored-by: Juelianqvq <Juelianqvq@noreply.github.com>

comaniac mentioned this pull request Sep 3, 2024

[Performance] Enable chunked prefill and prefix caching together #8120

Merged

triple-Mu pushed a commit to triple-Mu/vllm_official that referenced this pull request Sep 4, 2024

[Performance] Enable chunked prefill and prefix caching together (vll…

be60173

…m-project#7753)

comaniac mentioned this pull request Sep 10, 2024

[MISC] Keep chunked prefill enabled by default with long context when prefix caching is enabled #8342

Merged

Jeffwan pushed a commit to aibrix/vllm that referenced this pull request Sep 19, 2024

[Performance] Enable chunked prefill and prefix caching together (vll…

69eba29

…m-project#7753)

Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024

[Performance] Enable chunked prefill and prefix caching together (vll…

4b6fa2b

…m-project#7753) Signed-off-by: Alvant <alvasian@yandex.ru>

KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024

[Performance] Enable chunked prefill and prefix caching together (vll…

85def06

…m-project#7753)

[Performance] Enable chunked prefill and prefix caching together #7753

[Performance] Enable chunked prefill and prefix caching together #7753

Conversation

comaniac commented Aug 21, 2024 • edited Loading

github-actions bot commented Aug 21, 2024

rkooo567 commented Aug 21, 2024

sighingnow commented Aug 21, 2024 • edited Loading

comaniac commented Aug 21, 2024

sighingnow commented Aug 21, 2024

sighingnow commented Aug 21, 2024

comaniac commented Aug 21, 2024

sighingnow commented Aug 21, 2024 • edited Loading

comaniac commented Aug 21, 2024

comaniac commented Aug 21, 2024

sighingnow commented Aug 22, 2024 • edited Loading

rkooo567 left a comment

Choose a reason for hiding this comment

rkooo567 Aug 22, 2024

Choose a reason for hiding this comment

comaniac Aug 23, 2024

Choose a reason for hiding this comment

sighingnow commented Aug 22, 2024

comaniac commented Aug 22, 2024

JaheimLee commented Aug 22, 2024

comaniac commented Aug 22, 2024

Juelianqvq commented Aug 23, 2024

comaniac commented Aug 23, 2024

comaniac commented Aug 23, 2024

sighingnow commented Aug 24, 2024

comaniac commented Aug 24, 2024

comaniac commented Aug 26, 2024 • edited Loading

rkooo567 left a comment

Choose a reason for hiding this comment

rkooo567 Aug 28, 2024

Choose a reason for hiding this comment

comaniac Aug 28, 2024

Choose a reason for hiding this comment

rkooo567 Aug 28, 2024

Choose a reason for hiding this comment

comaniac Aug 28, 2024

Choose a reason for hiding this comment

comaniac commented Aug 28, 2024

Juelianqvq commented Aug 28, 2024

comaniac commented Aug 28, 2024

sighingnow commented Aug 29, 2024 • edited Loading

ashgold commented Sep 3, 2024

hmellor commented Sep 10, 2024

comaniac commented Sep 10, 2024

comaniac commented Aug 21, 2024 •

edited

Loading

sighingnow commented Aug 21, 2024 •

edited

Loading

sighingnow commented Aug 21, 2024 •

edited

Loading

sighingnow commented Aug 22, 2024 •

edited

Loading

comaniac commented Aug 26, 2024 •

edited

Loading

sighingnow commented Aug 29, 2024 •

edited

Loading