[Bugfix][Misc]: fix graph capture for decoder (vllm-project#9549)

Signed-off-by: Alvant <alvasian@yandex.ru>
compressa-ai · Oct 26, 2024 · 7fee2fa · 7fee2fa
1 parent e174d8c
commit 7fee2fa
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/vllm/worker/model_runner.py b/vllm/worker/model_runner.py
@@ -828,7 +828,7 @@ def build(self) -> ModelInputForGPU:
 
         cuda_graph_pad_size = self._get_cuda_graph_pad_size(
             num_seqs=len(seq_lens),
-            max_decode_seq_len=max_encoder_seq_len,
+            max_decode_seq_len=max_decode_seq_len,
             max_encoder_seq_len=max_encoder_seq_len)
 
         batch_size = len(input_tokens)