diff --git a/keps/2349-multikueue-external-custom-job-support/README.md b/keps/2349-multikueue-external-custom-job-support/README.md new file mode 100644 index 0000000000..31935e2da1 --- /dev/null +++ b/keps/2349-multikueue-external-custom-job-support/README.md @@ -0,0 +1,331 @@ +# KEP-2349: MultiKueue external custom Job support + + + + + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit Tests](#unit-tests) + - [Integration tests](#integration-tests) + - [Graduation Criteria](#graduation-criteria) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + + +## Summary + +This KEP introduces the ability to create Multikueue Admission Check controllers with custom set of adapters. + + + +## Motivation + +Many Kueue adaptors have company specific CustomResources. +Kueue should give a possibility to manage such CustomResources across multiple clusters. + + + +### Goals + +Allow specifying a controller and adapter for an external custom Job processed by Kueue. + + + +### Non-Goals + + + +## Proposal + + + +Make Multikueue admission check controller to run simultaneously with another set of Multkiueue adapters in one cluster. +Provide a custom admission check controller name and adapters for Multikueue. +Specified controller is used by workload, cluster and admission check reconcilers. + +### User Stories (Optional) + + + +#### Story 1 + +I would like to support the extension mechanism so that we can implement the MultiKueue controller for the custom / in-house Jobs. + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + + + +## Design Details + +Provide a way to integrate external controllers to manage Jobs in the Kueue. +MulikueueCluster API needs to be modified to ensure that one multikueue cluster is only managed by one admission check controller. + +1. Add ControllerName to the `MultiKueueClusterSpec` +```golang +type MultiKueueClusterSpec struct { + ... + // controllerName is name of the controller which will actually perform + // the checks. This is the name with which controller identifies with, + // not necessarily a K8S Pod or Deployment name. Cannot be empty. + // +kubebuilder:validation:Required + // +kubebuilder:default="kueue.x-k8s.io/multikueue" + // +kubebuilder:validation:XValidation:rule="self == oldSelf", message="field is immutable" + ControllerName string `json:"controllerName"` + ... +``` + +2. Make Controller Name used in Multikueue admission check configurable. +Extend MultiKueue Admission Check controller options to include controllerName. + +```golang +type SetupOptions struct { + ... + controllerName string + ... +} + +// WithControllerName - sets the controller name for which the multikueue +// admission check match. +func WithControllerName(controllerName string) SetupOption { + return func(o *SetupOptions) { + o.controllerName = controllerName + } +} + +func SetupControllers(mgr ctrl.Manager, namespace string, opts ...SetupOption) error { +... +} +``` + +3. Make the Adapters list for the MultiKueue Admission Check controller configurable. + +```golang +type SetupOptions struct { + ... + adapters map[string]jobframework.MultiKueueAdapter + ... +} + +// WithAdapters - sets all the MultiKueue adapters. +func WithAdapters(adapters map[string]jobframework.MultiKueueAdapter) SetupOption { + return func(o *SetupOptions) { + o.adapters = adapters + } +} + +// WithAdapter - sets or updates the adapter of the MultiKueue adapters. +func WithAdapter(adapter jobframework.MultiKueueAdapter) SetupOption { + return func(o *SetupOptions) { + o.adapters[adapter.GVK().String()] = adapter + } +} +``` + + + +### Test Plan + + + +[x] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +#### Unit Tests + + + +Target is to keep same level or increase the coverage for all of the modified packages. +Cover the critical part of newly created code. + + + +- ``: `` - `` + +#### Integration tests + + + +We are going to cover the following scenario: + +Setup 2 distinct MultiKueue Admission Checks over the same multicluster. +Each Admission Check with different set of adapters and different controller names. +Prove that 2 different MultiKueue controllers can manage different job types runnning successfully in parallel. + +### Graduation Criteria + + + +## Implementation History + +* 2024-06-20 First draft + + +## Drawbacks + + + +## Alternatives + + + +1. Dedicated API for MultiKueue communication between admission check controller and abstraction layer controller. + +2. Don't do any code changes, a custom controller can: +Import `multikueue` and `jobframework`, add adapters and run the MultKiueue admission check controller. +Will conflict with multikueue admission check controller in Kueue, which should be disabled by the feature gate in order to work. + +3. Instead of having MultiKueue cluster associated with controller name, change the Multikueue cluster status to use a custom active condition for old controllers (controller name) that use it. diff --git a/keps/2349-multikueue-external-custom-job-support/kep.yaml b/keps/2349-multikueue-external-custom-job-support/kep.yaml new file mode 100644 index 0000000000..a74c4b129b --- /dev/null +++ b/keps/2349-multikueue-external-custom-job-support/kep.yaml @@ -0,0 +1,30 @@ +title: MultiKueue external custom Job support +kep-number: 2349 +authors: + - "@mszadkow" +status: implementable +creation-date: 2024-06-20 +reviewers: + - "@tenzen-y" + - "@mimowo" + - "@trasc" +approvers: + - "@tenzen-y" + - "@mimowo" + +see-also: + - "/keps/693-multikueue" + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v0.8" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v0.8" + beta: "v0.8" +