Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[serve] default max_concurrent_queries is too high #42987

Closed
zcin opened this issue Feb 5, 2024 · 0 comments · Fixed by #45943
Closed

[serve] default max_concurrent_queries is too high #42987

zcin opened this issue Feb 5, 2024 · 0 comments · Fixed by #45943
Assignees
@zcin
Labels
ray-2.10 ray-2.11

Comments

@zcin
Copy link
Contributor

zcin commented Feb 5, 2024 •
edited
Loading

The default for max_concurrent_queries (100) is currently too high, we should lower it to 5 or 10. This default will work much better with the defaults for autoscaling as well.

  • Change the default for 2.22 and add a warning
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zcin zcin

Labels
ray-2.10 ray-2.11
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[serve] change default for max and target ongoing requests zcin/ray
1 participant
@zcin