Skip to content

Commit

Permalink
[BugFix][Core] Multistep Fix Crash on Request Cancellation (vllm-proj…
Browse files Browse the repository at this point in the history
  • Loading branch information
robertgshaw2-neuralmagic authored Aug 31, 2024
1 parent 4b7034f commit 2029ce5
Showing 1 changed file with 7 additions and 1 deletion.
8 changes: 7 additions & 1 deletion vllm/engine/output_processor/multi_step.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,15 @@ def process_outputs(self,
# TODO: Add support for async if necessary
assert not is_async

# Sequences can be in RUNNING or FINISHED_ABORTED state
# once scheduled, as a sequence is moved to FINSIHED_ABORTED
# if a client disconnects from the api server.
seqs = sequence_group.get_seqs(status=SequenceStatus.RUNNING)
if seqs is None:
seqs = sequence_group.get_seqs(
status=SequenceStatus.FINISHED_ABORTED)

assert seqs, "expected running sequences"
assert seqs, "Expected RUNNING or FINISHED_ABORTED sequences"
assert len(seqs) == 1, (
"Beam search not supported in multi-step decoding.")
seq = seqs[0]
Expand Down

0 comments on commit 2029ce5

Please sign in to comment.