Upgrade RayJob API to v1

[astefanutti]

What this PR does / why we need it:

KubeRay has promoted the RayJob API from v1alpha1 to v1 with KubeRay v1.0.0. While the v1alpha1 version is still served in KubeRay v1.1.0, it'll be removed in v1.2.0.

Which issue(s) this PR fixes:

Fixes #1335.

Special notes for your reviewer:

Supersedes #1435.

Does this PR introduce a user-facing change?

Upgrade RayJob API to v1

ACTION REQUIRED: If you use KubeRay older than v1.0.0, you'll have to upgrade your existing installation
to KubeRay v1.0.0, or any more recent version, that supports KubeRay v1 APIs, for it to
remain compatible with Kueue.

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue ready!

Name	Link
🔨 Latest commit	379e4ba
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/660c39ecf54efc0008ae9628
😎 Deploy Preview	https://deploy-preview-1802--kubernetes-sigs-kueue.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[mbzomowski]

Hey, apologies if I'm jumping the gun here, but I'm currently trying to get v1 RayJobs to work with Kueue - I pulled down your PR changes, and installed Kueue through helm, but I'm getting an error whenever I try to create a new RayJob now:

Error from server (InternalError): error when creating "ray-job.sample.yaml": Internal error occurred: failed calling webhook "mrayjob.kb.io": failed to call webhook: the server could not find the requested resource

Looking at the Kueue controller logs, it seems like the mutating (and I assume validating) webhook is still for v1alpha1 RayJobs:

{"level":"info","ts":"2024-03-12T22:31:38.10128149Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:158","msg":"Registering a mutating webhook","GVK":"ray.io/v1alpha1, Kind=RayJob","path":"/mutate-ray-io-v1alpha1-rayjob"}

Am I doing something wrong here? Possibly installing Kueue in the wrong way to utilize these changes?

[astefanutti]

Hey, apologies if I'm jumping the gun here, but I'm currently trying to get v1 RayJobs to work with Kueue - I pulled down your PR changes, and installed Kueue through helm, but I'm getting an error whenever I try to create a new RayJob now:

No pb at all :) Thanks a lot for testing this!

Looking at the Kueue controller logs, it seems like the mutating (and I assume validating) webhook is still for v1alpha1 RayJobs:

{"level":"info","ts":"2024-03-12T22:31:38.10128149Z","logger":"controller-runtime.builder","caller":"builder/webhook.go:158","msg":"Registering a mutating webhook","GVK":"ray.io/v1alpha1, Kind=RayJob","path":"/mutate-ray-io-v1alpha1-rayjob"}

Am I doing something wrong here? Possibly installing Kueue in the wrong way to utilize these changes?

That "GVK":"ray.io/v1alpha1, Kind=RayJob" really looks like it should not be there if the changes from that PR would be taken into account, specifically: https://github.com/kubernetes-sigs/kueue/pull/1802/files#diff-24b8d3ba7104004ba575becf8db0a4ee47c9062f716a744f1afbbdb7bfded0d9R50

How exactly it is you've tested it? Could you double check the deployment uses the container image that's built with the changes from that PR?

[mbzomowski]

Ah whoops - realizing now that just running helm install ... after pulling down your changes wasn't enough.

I ran make install deploy, but now I'm getting the following error:

  Warning  Failed     14s (x2 over 29s)  kubelet            Failed to pull image "gcr.io/k8s-staging-kueue/kueue:v0.7.0-devel-33-g0414b335": rpc error: code = NotFound desc = failed to pull and unpack image "gcr.io/k8s-staging-kueue/kueue:v0.7.0-devel-33-g0414b335": failed to resolve reference "gcr.io/k8s-staging-kueue/kueue:v0.7.0-devel-33-g0414b335": gcr.io/k8s-staging-kueue/kueue:v0.7.0-devel-33-g0414b335: not found

I'm guessing I should build the new image locally and use that? Are there instructions for testing anywhere? Is make install deploy enough to try out the changes once I get the image built?

[mbzomowski]

Oof nevermind, sorry just found this: https://kueue.sigs.k8s.io/docs/installation/#build-and-install-from-source

I'll try this again

[mbzomowski]

Ok, I built & installed from source and confirmed I'm now getting the v1 RayJob mutating & admission webhooks. kubectl -f apply'd a RayJob, and the RayJob gets created, and so does the workload, but nothing else. And I'm seeing this in the kueue-controller logs:

{"level":"error","ts":"2024-03-13T17:27:04.793883358Z","logger":"cert-rotation","caller":"rotator/rotator.go:322","msg":"could not refresh CA and server certs","error":"Operation cannot be fulfilled on secrets \"kueue-webhook-server-cert\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded.func1\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.1/pkg/rotator/rotator.go:322\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:461\ngit.luolix.top/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.1/pkg/rotator/rotator.go:350\ngit.luolix.top/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).Start\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.1/pkg/rotator/rotator.go:278\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:223"}

[astefanutti]

Ok, I built & installed from source and confirmed I'm now getting the v1 RayJob mutating & admission webhooks. kubectl -f apply'd a RayJob, and the RayJob gets created, and so does the workload, but nothing else. And I'm seeing this in the kueue-controller logs:

{"level":"error","ts":"2024-03-13T17:27:04.793883358Z","logger":"cert-rotation","caller":"rotator/rotator.go:322","msg":"could not refresh CA and server certs","error":"Operation cannot be fulfilled on secrets \"kueue-webhook-server-cert\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded.func1\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.1/pkg/rotator/rotator.go:322\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:461\ngit.luolix.top/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.1/pkg/rotator/rotator.go:350\ngit.luolix.top/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).Start\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.1/pkg/rotator/rotator.go:278\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:223"}

That error message happens when optimistic locking fails during an update, but the operation is simply retried so it's not an issue.

For the RayJob and the associated workload resource, checking their statuses should provide the reason why it's not started / running.

[mbzomowski]

RayJob is stuck in Initializing status:

Status:
  Job Deployment Status:  Initializing
  Job Id:                 rayjob-sample-mpmnf
  Ray Cluster Name:       rayjob-sample-raycluster-c2hgb
  Ray Cluster Status:
    Desired CPU:     0
    Desired GPU:     0
    Desired Memory:  0
    Desired TPU:     0
    Head:
  Start Time:  2024-03-13T19:43:01Z

Workload status doesn't show anything out of the ordinary:

  Conditions:
    Last Transition Time:  2024-03-13T19:43:01Z
    Message:               Quota reserved in ClusterQueue cluster-queue
    Reason:                QuotaReserved
    Status:                True
    Type:                  QuotaReserved
    Last Transition Time:  2024-03-13T19:43:01Z
    Message:               The workload is admitted
    Reason:                Admitted
    Status:                True
    Type:                  Admitted

RayCluster is showing as suspended, though:

  Normal  Deleted  4m (x4 over 9m)  raycluster-controller  Deleted Pods for RayCluster default/rayjob-sample-raycluster-c2hgb due to suspension

Also seeing these errors in the kueue-controller logs (in addition to that other one above):

{"level":"error","ts":"2024-03-13T19:41:54.423509864Z","caller":"jobframework/reconciler.go:202","msg":"Removing finalizer","controller":"rayjob","controllerGroup":"ray.io","controllerKind":"RayJob","RayJob":{"name":"rayjob-sample","namespace":"default"},"namespace":"default","name":"rayjob-sample","reconcileID":"c3bc5a6e-4af2-46a9-9656-939200d808b5","job":"default/rayjob-sample","gvk":"ray.io/v1, Kind=RayJob","error":"Operation cannot be fulfilled on workloads.kueue.x-k8s.io \"rayjob-rayjob-sample-8f26e\": StorageError: invalid object, Code: 4, Key: /registry/kueue.x-k8s.io/workloads/default/rayjob-rayjob-sample-8f26e, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4c144f8e-b609-4495-84c1-3a71a73dc4d6, UID in object meta: ","stacktrace":"sigs.k8s.io/kueue/pkg/controller/jobframework.(*JobReconciler).ReconcileGenericJob\n\t/workspace/pkg/controller/jobframework/reconciler.go:202\nsigs.k8s.io/kueue/pkg/controller/jobframework.(*genericReconciler).Reconcile\n\t/workspace/pkg/controller/jobframework/reconciler.go:1003\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-03-13T19:41:54.423578204Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"rayjob","controllerGroup":"ray.io","controllerKind":"RayJob","RayJob":{"name":"rayjob-sample","namespace":"default"},"namespace":"default","name":"rayjob-sample","reconcileID":"c3bc5a6e-4af2-46a9-9656-939200d808b5","error":"Operation cannot be fulfilled on workloads.kueue.x-k8s.io \"rayjob-rayjob-sample-8f26e\": StorageError: invalid object, Code: 4, Key: /registry/kueue.x-k8s.io/workloads/default/rayjob-rayjob-sample-8f26e, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4c144f8e-b609-4495-84c1-3a71a73dc4d6, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/controller/controller.go:227"}

[astefanutti]

@mbzomowski I've been able to reproduce. That issue is actually unrelated to that PR and should be fixed with #1844.

[mbzomowski]

Rebased with the new changes & everything works! Thanks so much @astefanutti!

[astefanutti]

@mbzomowski great, thanks for the feedback, and the thorough testing!

[trasc]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: eeb318d227895819982a137aae623f27907664e9

[astefanutti]

@alculquicondor @tenzen-y @andrewsykim @kevin85421 Do we want to get this in the next Kueue minor release, now that KubeRay v1.1.0 has been released?

[tenzen-y]

@alculquicondor @tenzen-y @andrewsykim @kevin85421 Do we want to get this in the next Kueue minor release, now that KubeRay v1.1.0 has been released?

Based on this discussion: #1435 (comment)

I agree with the migration of KubeRay v1 in Kueue v0.7.

[alculquicondor]

sgtm, please rebase

And put ACTION REQUIRED in the release notes to ask users to upgrade to a version of kuberay that supports v1 (1.0?)

[astefanutti]

PR rebased, and release notes updated. PTAL, thanks.

[tenzen-y]

@astefanutti Could you address CI errors?

[astefanutti]

/retest

[astefanutti]

@tenzen-y retesting as I think these were transient failures.

[tenzen-y]

Thank you!
/lgtm
/approve

@astefanutti Could you update documentation and examples in a follow-up?

• https://github.com/kubernetes-sigs/kueue/blob/main/site/content/en/docs/tasks/run/rayjobs.md
• https://github.com/kubernetes-sigs/kueue/blob/main/site/static/examples/jobs/ray-job-sample.yaml

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 261fce03970d3454a8a9b35fb6516b8050daff94

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astefanutti, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[astefanutti]

@astefanutti Could you update documentation and examples in a follow-up?

• https://github.com/kubernetes-sigs/kueue/blob/main/site/content/en/docs/tasks/run/rayjobs.md
• https://github.com/kubernetes-sigs/kueue/blob/main/site/static/examples/jobs/ray-job-sample.yaml

@tenzen-y as far as I can see, the documentation and examples have been updated in the scope of that PR already, unless you have specific changes in mind?

On the other, while giving it another look, I've found another follow-up that's covered by #1941.

[andrewsykim]

Sorry I'm late to review this, @astefanutti do we also want to update the go module version of kuberay? I ran into some corner cases where webhook logic drops fields because the imported type in Kueue doesn't have all the needed fields.

[andrewsykim]

We should update to v1.1, but make sure it's backwards compatible with a cluster on v1.0

[andrewsykim]

Oh ignore me, looks like it's using v1.1 already :)

[tenzen-y]

@astefanutti Could you update documentation and examples in a follow-up?

• https://github.com/kubernetes-sigs/kueue/blob/main/site/content/en/docs/tasks/run/rayjobs.md
• https://github.com/kubernetes-sigs/kueue/blob/main/site/static/examples/jobs/ray-job-sample.yaml

@tenzen-y as far as I can see, the documentation and examples have been updated in the scope of that PR already, unless you have specific changes in mind?

On the other, while giving it another look, I've found another follow-up that's covered by #1941.

Oops, you're right.
NVM

[astefanutti]

Oops, you're right. NVM

No pb, just wanted to make sure I was not missing anything!