Add support for RayCluster

[andrewsykim]

What would you like to be added:

Support RayCluster as a queue-able workload in Kueue (much like RayJob).

Why is this needed:

Currently Kueue supports RayJob which works great when managing ray jobs that run on ephemeral ray clusters. However, there are many use-cases and existing workloads that depend on long-lived RayClusters. Being able to account for these RayClusters with Kueue would greatly improve integration of Kueue with Ray.

Completion requirements:

This probably needs a KEP, but very roughly the requirements would be:

• Kueue should only account for ray worker nodes in ClusterQueue quotas (we may account for head nodes later)
• Scaling of worker nodes should be blocked on quota reservations
• Evicting or pre-empting RayClusters involves scaling worker groups to 0

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[andrewsykim]

@alculquicondor initial thoughts on this? I can try to put together a KEP if this idea doesn't sound crazy :)

[trasc]

Hi, in my opinion an easy way to do this is to have Ray create "Kueueble" pods for the workers.

[alculquicondor]

It sounds reasonable to me to add support for RayCluster.
The additional thing to take into consideration is to make sure RayJobs are not doubly queued once RayCluster is also queueable.

cc @kerthcet as one of the primary users of Ray+Kueue

[tenzen-y]

+1

I think that supporting long-living resources would be worth it.

[kerthcet]

Make sense to me, several points here:

• RayCluster is somehow similar to RayJob because RayJob reuses rayClusterSpec, so the code should be reuse.
• RayCluster doesn't support suspend semantics, so it behaves as long-lived resources, right? Then we can't guarantee the resource utilization if the cluster is empty of jobs.

make sure RayJobs are not doubly queued once RayCluster is also queueable.

I think we can tell this via the clusterSelector field.

[andrewsykim]

/assign

[andrewsykim]

Do we still want a KEP for this? I think with the new suspend API (ray-project/kuberay#1667), the implementation should be very straight forward.

[alculquicondor]

Historically, we haven't written KEPs for integrations.

FWIIW, the implementation should be quite similar to that of Job, whereas the implementation for RayJob will change a little to be more similar to Job. Note that if a user is queuing a RayJob, its RayCluster shouldn't be doubly queued.

[andrewsykim]

FYI @vicentefb who is working on the feature now