Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support more optioin about usage in stream mode #985

Merged
merged 12 commits into from
Aug 8, 2024

Conversation

yichuan520030910320
Copy link
Collaborator

@yichuan520030910320 yichuan520030910320 commented Aug 8, 2024

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Fix #954

Modification

adapter and protocol in openai

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

@Ying1123
Copy link
Member

Ying1123 commented Aug 8, 2024

What is wrong with the default mem_fraction_static?

@meetzuber
Copy link

I hope all finish reasons are considered as per openai api format.
Please refer below link for more details.
https://platform.openai.com/docs/guides/chat-completions/response-format


Every response will include a finish_reason. The possible values for finish_reason are:
stop: API returned complete message, or a message terminated by one of the stop sequences provided via the stop parameter
length: Incomplete model output due to max_tokens parameter or token limit
function_call: The model decided to call a function
content_filter: Omitted content due to a flag from our content filters
null: API response still in progress or incomplete
Depending on input parameters, the model response may include different information.

python/sglang/srt/openai_api/adapter.py Outdated Show resolved Hide resolved
test/srt/test_openai_server.py Show resolved Hide resolved
@yichuan520030910320
Copy link
Collaborator Author

@meetzuber please stay tuned for the function_call and content filter, we will support that in the future

Copy link
Contributor

@merrymercy merrymercy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please resolve the conflicts and we can merge this

@merrymercy merrymercy enabled auto-merge (squash) August 8, 2024 09:01
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
@merrymercy merrymercy merged commit 3a79613 into sgl-project:main Aug 8, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] usage is null when set stream=True
4 participants