Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] Potential performance regression during ray start due to new RPC #38509

Closed
vitsai opened this issue Aug 16, 2023 · 3 comments
Closed

[core] Potential performance regression during ray start due to new RPC #38509

vitsai opened this issue Aug 16, 2023 · 3 comments
Assignees
@vitsai
Labels
core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release

Comments

@vitsai
Copy link
Contributor

vitsai commented Aug 16, 2023

A new blocking RPC to GCS was added to ray start in #37583. Measure the regression and optimize if necessary.
@vitsai vitsai added the P1 Issue that should be fixed within a few weeks label Aug 16, 2023
@vitsai vitsai self-assigned this Aug 16, 2023
@scv119 scv119 added the core Issues that should be addressed in Ray Core label Aug 16, 2023
@scv119
Copy link
Contributor

scv119 commented Aug 16, 2023

cc @iycheng can you add a bit more details for the repro?

@vitsai vitsai added P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release and removed P1 Issue that should be fixed within a few weeks labels Aug 21, 2023
@vitsai
Copy link
Contributor Author

vitsai commented Aug 23, 2023

Can check this with ray start --head.

@vitsai
Copy link
Contributor Author

vitsai commented Aug 23, 2023

Root cause is the newly added blocking RPC when initializing the GCS client during ray start, which takes 5 seconds and occurs before all other processes are started. This RPC basically serializes the rest of ray start after GCS server completely starts.

@vitsai vitsai mentioned this issue Aug 25, 2023
Merged
8 tasks
@scv119 scv119 removed their assignment Aug 28, 2023
@vitsai vitsai closed this as completed Sep 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vitsai vitsai

Labels
core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@scv119 @vitsai