Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support a custom logit processor #2291

Open
merrymercy opened this issue Dec 1, 2024 · 4 comments
Open

[Feature] Support a custom logit processor #2291

merrymercy opened this issue Dec 1, 2024 · 4 comments
Labels
good first issue Good for newcomers help wanted Extra attention is needed

Comments

@merrymercy
Copy link
Contributor

merrymercy commented Dec 1, 2024

We would like to support custom logit processors registered by users, so users can easily implement their custom sampling methods without the need to change the sglang code.

Steps

  1. After the following line, allowing running some user code to process the logits
    logits = logits.contiguous()
  2. Add a new field custom_params to SamplingParams
    skip_special_tokens: bool = True,
    so users can pass in their arguments.

See also huggingface/transformers#14779

@merrymercy merrymercy added good first issue Good for newcomers help wanted Extra attention is needed labels Dec 1, 2024
@zhyncs
Copy link
Member

zhyncs commented Dec 1, 2024

BTW, constrained decoding and function calling can also be implemented this way, such as by crafting a mask kernel.
The schema part can continue to use outlines or xgrammar.

@zhyncs
Copy link
Member

zhyncs commented Dec 1, 2024

@zhyncs
Copy link
Member

zhyncs commented Dec 1, 2024

cc @Ubospica

@hongpeng-guo
Copy link

May I take this issue if no one is looking at it rn?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants