[WIP] Add sampler logit processor

[hongpeng-guo]

Motivation

This PR tries to support custom logit processors registered by users, so users can easily implement their custom sampling methods without the need to change the sglang code.

Related Issue

#2291

Modifications

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[merrymercy]

Thanks for taking this. Can you add some end-to-end tests and examples?

[hongpeng-guo]

Thanks for taking this. Can you add some end-to-end tests and examples?

@merrymercy Thanks for taking a look on this. I am still trying to understand the appropriate layer for the user to register their customized_logit_processor. The goal is to enable customized_logit_processor functionality without requiring changes to the internal sglang codebase. To achieve this, the function registration should occur at the API layer. The customized_logit_processor_fn and custom_params will then be passed from the program driver to the internal modules, such as Sampler and SampleParams.

If the above seems correct, I will try to get it reviewable within this week.