[Bugfix][Misc]: fix graph capture for decoder (vllm-project#9549)

Signed-off-by: charlifu <charlifu@amd.com>
charlifu · Oct 23, 2024 · 48bc888 · 48bc888
1 parent 60626e7
commit 48bc888
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/vllm/worker/model_runner.py b/vllm/worker/model_runner.py
@@ -828,7 +828,7 @@ def build(self) -> ModelInputForGPU:
 
         cuda_graph_pad_size = self._get_cuda_graph_pad_size(
             num_seqs=len(seq_lens),
-            max_decode_seq_len=max_encoder_seq_len,
+            max_decode_seq_len=max_decode_seq_len,
             max_encoder_seq_len=max_encoder_seq_len)
 
         batch_size = len(input_tokens)