RayCluster integration with Kueue #1520
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
do-not-merge/work-in-progress
✅ Deploy Preview for kubernetes-sigs-kueue canceled.
|
k8s-ci-robot
added
needs-ok-to-test
|
Hi @vicentefb. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
the
size/XXL
|
Will squash commits once final review is given. |
vicentefb
force-pushed
the
KueueRayCluster
branch
from
32d1e36 to
1fb4a7c
Compare
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
andrewsykim
reviewed
Dec 27, 2023
test/integration/controller/jobs/raycluster/raycluster_controller_test.go
Outdated
Show resolved
Hide resolved
test/integration/controller/jobs/raycluster/raycluster_controller_test.go
Outdated
Show resolved
Hide resolved
test/integration/controller/jobs/raycluster/raycluster_controller_test.go
Outdated
Show resolved
Hide resolved
vicentefb
force-pushed
the
KueueRayCluster
branch
2 times, most recently
from
c59763f to
79b174a
Compare
|
/hold Holding until there's a kuberay release that has ray-project/kuberay#1711 |
k8s-ci-robot
added
the
do-not-merge/hold
|
/retest |
vicentefb
force-pushed
the
KueueRayCluster
branch
3 times, most recently
from
673c99c to
0765c64
Compare
vicentefb
changed the title
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
vicentefb
force-pushed
the
KueueRayCluster
branch
from
13b200f to
24e1dd5
Compare
|
So it doesn't currently work? I'm rather opposed to merging without it. /hold |
Comment on lines
+197
to
+200
| gomega.Expect(len(createdJob.Spec.HeadGroupSpec.Template.Spec.NodeSelector)).Should(gomega.Equal(1)) | ||
| gomega.Expect(createdJob.Spec.HeadGroupSpec.Template.Spec.NodeSelector[instanceKey]).Should(gomega.Equal(onDemandFlavor.Name)) | ||
| gomega.Expect(len(createdJob.Spec.WorkerGroupSpecs[0].Template.Spec.NodeSelector)).Should(gomega.Equal(1)) | ||
| gomega.Expect(createdJob.Spec.WorkerGroupSpecs[0].Template.Spec.NodeSelector[instanceKey]).Should(gomega.Equal(spotFlavor.Name)) |
There was a problem hiding this comment.
ok, it is tested here. It would be useful to have this in unit tests, but it's ok to leave for a follow up
cc @astefanutti
/hold cancel.
astefanutti
reviewed
Jan 25, 2024
| }{ | ||
| "when workload is admitted, cluster is unsuspended": { | ||
| job: *baseJobWrapper.Clone(). | ||
| NodeSelectorHeadGroup("provisioning", "spot"). |
There was a problem hiding this comment.
Should this should be removed from the test input, as it's expected to be added during the reconciliation?
There was a problem hiding this comment.
Also a flavor needs to be defined, because that's where the selectors are obtained from.
| }{ | ||
| "when workload is admitted, cluster is unsuspended": { | ||
| job: *baseJobWrapper.Clone(). | ||
| NodeSelectorHeadGroup("provisioning", "spot"). |
There was a problem hiding this comment.
Also a flavor needs to be defined, because that's where the selectors are obtained from.
vicentefb
force-pushed
the
KueueRayCluster
branch
4 times, most recently
from
f8c09fb to
916328e
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alculquicondor, astefanutti, vicentefb The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm based on #1520 (review) |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 4ec47e5614815d8b5ed50a02d7a1e3b40c34ba50
|
|
/hold cancel |
k8s-ci-robot
removed
the
do-not-merge/hold
|
The periodic E2E tests are healthy... so could it be that this PR is breaking them? /test pull-kueue-test-e2e-main-1-26 |
updated kubebuilder markers but still workload isnt created updating branch update unit tests working ? changes ray cluster admitted as a workload and running, it has a lot of debug log lines that need to be removed added comma fixed helper method, need to implemente tests changes v1alpha1 to v1 but still workload isnt created pods are not getting suspended :( charts WIP admit RayCluster as kueuable workload updated kubebuilder markers but still workload isnt created updating branch update unit tests working ? changes ray cluster admitted as a workload and running, it has a lot of debug log lines that need to be removed fixed helper method, need to implemente tests changes v1alpha1 to v1 but still workload isnt created updated wrappers not all tests are passing but still working on them debugging podReady test update updated controller and webhook tests, now working updated raycluster webhook test added ray cluster sample yaml updated go modules since this is using kuberay masters version removing diffs from reconciler file removing diffs from reconciler file removing diffs from reconciler file updated role yaml file added TODO comment for autoscaler updated raycluster controller test added scheme for v1 inside register file update rayjob import to reference v1 otherwise tests are not passing in PR changed the order of jobs list updated pod controller api version to be v1 instead of v1alpha1 update go files fixed pull kueue test reverted changes made to rayjob import library version fixed rayjob tests WIP admit RayCluster as kueuable workload updated kubebuilder markers but still workload isnt created updating branch update unit tests working ? changes ray cluster admitted as a workload and running, it has a lot of debug log lines that need to be removed added comma fixed helper method, need to implemente tests changes v1alpha1 to v1 but still workload isnt created pods are not getting suspended :( charts WIP admit RayCluster as kueuable workload updated kubebuilder markers but still workload isnt created updating branch update unit tests working ? changes ray cluster admitted as a workload and running, it has a lot of debug log lines that need to be removed fixed helper method, need to implemente tests changes v1alpha1 to v1 but still workload isnt created updated wrappers not all tests are passing but still working on them debugging podReady test update updated controller and webhook tests, now working updated raycluster webhook test updated go modules since this is using kuberay masters version removing diffs from reconciler file removing diffs from reconciler file removing diffs from reconciler file updated role yaml file added TODO comment for autoscaler updated raycluster controller test added scheme for v1 inside register file update rayjob import to reference v1 otherwise tests are not passing in PR changed the order of jobs list updated pod controller api version to be v1 instead of v1alpha1 update go files fixed pull kueue test reverted changes made to rayjob import library version fixed rayjob tests updated example raycluster nit updated ray cluster controller unit test and wrapper updated tests and charts updated ownerReference for rayJob and rayCluster removed extra configuration for pods and duplicated text generated by script added third argument for reconciler
reverted git tag change moved the sample yaml file to a different branch addressed comments and used generalised method call in reconciler to check ownership updated new reconciler variable addressed comments nit removed register changes revert changes to go dependencies to test something updated go files nit added schema nit added files generated by make verify update charts test test updated register test fixed test added builder n update fix go modules fixed test nit added case for ray cluster completion nit debugging added test for coverage updated the restore node test nit addressed comments update added scheme for rayv1 in tests nit updated test to inject node selector with flavor defined nit nit
vicentefb
force-pushed
the
KueueRayCluster
branch
from
916328e to
c9080e3
Compare
k8s-ci-robot
removed
the
lgtm
|
tests are definitely flaky #1658 |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 9882d57638bb7418ba59d98bf166f1829eff6ef9
|
kannon92
pushed a commit
to openshift-kannon92/kubernetes-sigs-kueue
that referenced
this pull request
Nov 19, 2024
* WIP admit RayCluster as kueuable workload updated kubebuilder markers but still workload isnt created updating branch update unit tests working ? changes ray cluster admitted as a workload and running, it has a lot of debug log lines that need to be removed added comma fixed helper method, need to implemente tests changes v1alpha1 to v1 but still workload isnt created pods are not getting suspended :( charts WIP admit RayCluster as kueuable workload updated kubebuilder markers but still workload isnt created updating branch update unit tests working ? changes ray cluster admitted as a workload and running, it has a lot of debug log lines that need to be removed fixed helper method, need to implemente tests changes v1alpha1 to v1 but still workload isnt created updated wrappers not all tests are passing but still working on them debugging podReady test update updated controller and webhook tests, now working updated raycluster webhook test added ray cluster sample yaml updated go modules since this is using kuberay masters version removing diffs from reconciler file removing diffs from reconciler file removing diffs from reconciler file updated role yaml file added TODO comment for autoscaler updated raycluster controller test added scheme for v1 inside register file update rayjob import to reference v1 otherwise tests are not passing in PR changed the order of jobs list updated pod controller api version to be v1 instead of v1alpha1 update go files fixed pull kueue test reverted changes made to rayjob import library version fixed rayjob tests WIP admit RayCluster as kueuable workload updated kubebuilder markers but still workload isnt created updating branch update unit tests working ? changes ray cluster admitted as a workload and running, it has a lot of debug log lines that need to be removed added comma fixed helper method, need to implemente tests changes v1alpha1 to v1 but still workload isnt created pods are not getting suspended :( charts WIP admit RayCluster as kueuable workload updated kubebuilder markers but still workload isnt created updating branch update unit tests working ? changes ray cluster admitted as a workload and running, it has a lot of debug log lines that need to be removed fixed helper method, need to implemente tests changes v1alpha1 to v1 but still workload isnt created updated wrappers not all tests are passing but still working on them debugging podReady test update updated controller and webhook tests, now working updated raycluster webhook test updated go modules since this is using kuberay masters version removing diffs from reconciler file removing diffs from reconciler file removing diffs from reconciler file updated role yaml file added TODO comment for autoscaler updated raycluster controller test added scheme for v1 inside register file update rayjob import to reference v1 otherwise tests are not passing in PR changed the order of jobs list updated pod controller api version to be v1 instead of v1alpha1 update go files fixed pull kueue test reverted changes made to rayjob import library version fixed rayjob tests updated example raycluster nit updated ray cluster controller unit test and wrapper updated tests and charts updated ownerReference for rayJob and rayCluster removed extra configuration for pods and duplicated text generated by script added third argument for reconciler * added logic to ignore rayclusters created by a RayJob reverted git tag change moved the sample yaml file to a different branch addressed comments and used generalised method call in reconciler to check ownership updated new reconciler variable addressed comments nit removed register changes revert changes to go dependencies to test something updated go files nit added schema nit added files generated by make verify update charts test test updated register test fixed test added builder n update fix go modules fixed test nit added case for ray cluster completion nit debugging added test for coverage updated the restore node test nit addressed comments update added scheme for rayv1 in tests nit updated test to inject node selector with flavor defined nit nit
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
What this PR does / why we need it:
Support RayCluster as a queue-able workload in Kueue since there are many use-cases and existing workloads that depend on long-lived RayClusters.
Which issue(s) this PR fixes:
Fixes #1272
Special notes for your reviewer:
This implementation is using master's version of kuberay and using
v1api version instead ofv1alpha1because it's using the new suspend API (ray-project/kuberay#1667) for RayClusterDoes this PR introduce a user-facing change?