-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ray TPU Webhook Reliability Improvements #723
Merged
ryanaoleary
merged 61 commits into
GoogleCloudPlatform:main
from
ryanaoleary:autoscaling-changes
Jul 25, 2024
Merged
Ray TPU Webhook Reliability Improvements #723
ryanaoleary
merged 61 commits into
GoogleCloudPlatform:main
from
ryanaoleary:autoscaling-changes
Jul 25, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrewsykim
reviewed
Jul 16, 2024
ray-on-gke/tpu/kuberay-tpu-webhook/deployments/validating-webhook-cfg.yaml
Show resolved
Hide resolved
andrewsykim
reviewed
Jul 16, 2024
andrewsykim
reviewed
Jul 17, 2024
andrewsykim
reviewed
Jul 17, 2024
andrewsykim
reviewed
Jul 18, 2024
andrewsykim
reviewed
Jul 19, 2024
andrewsykim
reviewed
Jul 19, 2024
2 tasks
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
andrewsykim
reviewed
Jul 24, 2024
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
67cc91c
to
f9e1b4f
Compare
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
f9e1b4f
to
54c97f1
Compare
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
andrewsykim
reviewed
Jul 24, 2024
andrewsykim
reviewed
Jul 24, 2024
andrewsykim
reviewed
Jul 24, 2024
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
andrewsykim
reviewed
Jul 24, 2024
andrewsykim
reviewed
Jul 24, 2024
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
andrewsykim
reviewed
Jul 25, 2024
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
andrewsykim
approved these changes
Jul 25, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR depends on changes from #740 which should be merged first (c4cdf31 marks the start of the changes in this PR).
This PR improves the reliability of the webhook by making it stateless in between calls, fixing issues related to the
sliceToWorkers
mapping being cleared upon webhook restart. These changes rely on adding a k8s PodInformer to the webhook that watches Pods in the GKE cluster with theray.io/node-type=worker
label. We can then determine the nextreplicaIndex
andTPU_WORKER_ID
using the PodInformer cache. These changes remove the need to intercept Pod deletion requests.This PR has been tested as follows: