kubeflow/spark-operator 'SparkApplication' integration

[everpeace]

• This PR depends on unmerged depdendency: Suspend/Resume feature on SparkApplication kubeflow/spark-operator#2387
• Controller and its unit tests
• Webhook and its unit test
• Integration Tests
• E2E Tests (As no custom job controllers implements E2E tests)
• Write Documents

---

What type of PR is this?

/kind feature

What this PR does / why we need it:

This adds kubeflow/spark-operator's SparkApplication integration in Kueue.

Technically, Kueue's Plain Pods integration can handle SparkApplication pods by putting kueue.x-k8s.io/* labels both in driver/executor pod template. However, SparkApplication integration can give simpler, better UX and deeper integrations for some Kueue's unique features: Topology Aware Scheduling, All-or-Nothing Scheduling, etc.

Which issue(s) this PR fixes:

N/A

Special notes for your reviewer:

• Why kubeflow/spark-operator??
  • I know there are some other spark operators (e.g. https://github.com/apache/spark-kubernetes-operator). However, kubeflow's one is the most widely adopted one in my observation (bunch of literature for running this operator in public cloud's mangged kubernetes, etc.).

Does this PR introduce a user-facing change?

Support kubeflow/spark-operator SparkApplication integration

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: everpeace
Once this PR has been reviewed and has the lgtm label, please assign mimowo for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	6f7a6ca
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/6797042eb784cb0008502f8a

[everpeace]

I notice spark integration design was already proposed and approved in:
https://github.com/kubernetes-sigs/kueue/tree/main/keps/976-plain-pods#groups-of-pods-where-driver-generates-workers

This PR's design is far from the KEP. Thus, I'm closing this KEP.

@tenzen-y Thank you very much for this information.