Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Move ScheduleBatch out of SamplingInfo #1556

Merged
merged 3 commits into from
Oct 3, 2024
Merged

[Fix] Move ScheduleBatch out of SamplingInfo #1556

merged 3 commits into from
Oct 3, 2024

Conversation

merrymercy
Copy link
Contributor

@merrymercy merrymercy commented Oct 2, 2024

We can get regex info during get_model_worker_batch, so we do not need to keep a reference of ScheduleBatch in SamplingInfo

@merrymercy merrymercy merged commit 317631c into main Oct 3, 2024
10 of 11 checks passed
@merrymercy merrymercy deleted the fix-regex branch October 3, 2024 00:18
@qeternity
Copy link
Contributor

This seems to have broken constrained generation:

  File "/root/sglang/sglang/python/sglang/srt/model_executor/model_runner.py", line 520, in sample
    sampling_info.update_regex_vocab_mask()
  File "/root/sglang/sglang/python/sglang/srt/sampling/sampling_batch_info.py", line 123, in update_regex_vocab_mask
    self.vocab_mask[i].fill_(1)
    ~~~~~~~~~~~~~~~^^^
IndexError: index 2 is out of bounds for dimension 0 with size 2

Killed
[rank0]:W1004 19:16:06.465000 133394662926016 torch/_inductor/compile_worker/subproc_pool.py:126] SubprocPool unclean exit
/usr/lib/python3.11/multiprocessing/resource_tracker.py:123: UserWarning: resource_tracker: process died unexpectedly, relaunching.  Some resources might leak.
  warnings.warn('resource_tracker: process died unexpectedly, '
Traceback (most recent call last):
  File "/usr/lib/python3.11/multiprocessing/resource_tracker.py", line 239, in main
    cache[rtype].remove(name)
KeyError: '/mp-ieivjla9'

@merrymercy
Copy link
Contributor Author

Fixed by #1634

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants