-
-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add max_completions_tokens for o1 series models #857
add max_completions_tokens for o1 series models #857
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #857 +/- ##
==========================================
+ Coverage 98.46% 99.03% +0.57%
==========================================
Files 24 26 +2
Lines 1364 1452 +88
==========================================
+ Hits 1343 1438 +95
+ Misses 15 8 -7
Partials 6 6 ☔ View full report in Codecov by Sentry. |
@sashabaranov please take a look to the PR 🙃 |
@chococola great catch, thank you for the PR! Could we also panic when o1 is used and max_tokens≠0? |
Or just return an error, panic might be too much :) |
panic would be to much, and propably break backwards capabilities for oldest models. Maybe we should mark attribute as deprecated ? or leave the comment?
in all places ? and what about batch requests ? |
@chococola let's cover the most frequent chat completion + streaming case, all other cases would eventually be covered by other PRs :D |
4d313aa
to
be774ca
Compare
@sashabaranov i add validation check for |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, o1 is a can of worms! Thank you so much for looking so deeply into this! 🙌🏻
Describe the change
For new
o1 series models
attributemax_tokens
not working. In documentation this field marks deprecated.OpenAi introduced new attribute
max_completion_tokens
.Provide OpenAI documentation link
https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_tokens
https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_completion_tokens
https://platform.openai.com/docs/guides/reasoning/controlling-costs
https://community.openai.com/t/why-was-max-tokens-changed-to-max-completion-tokens/938077
Describe your solution
Update
ChatCompletionRequest
model