Mark Scheduler as deprecated

[thisthat]

Goal

Mark the scheduler as deprecated

Details

The operator can use the client to discover the K8s cluster version (see https://github.com/kubernetes/client-go/blob/7a7ea209369275ab8374bf7e89ea4fad99e820d7/discovery/discovery_client.go#L619).
Based on this, it can change the default value of the FF klt.readiness-gate. If the version supports the readiness gate in stable version, the FF default value is true; false otherwise.
The version of K8s we are looking for should be configured via Env Var and Helm Chart value.yaml and default to 1.27.

Similarly, the Helm chart should check for the Kubernetes version to decide whether the scheduler should be deployed.

DoD

• Based on K8s version, Helm chart doesn't add the scheduler
• Based on K8s version, the operator uses the Readiness Gate APIs or the Scheduler
• Documentation is updated

Depends on

• Adapt WorkloadInstance controller to support ReadinessGates #1477

[agardnerIT]

Does this have any user impact (eg. change in a way of working) or is this a "behind hte scenes" detail?

[RealAnna]

Does this have any user impact (eg. change in a way of working) or is this a "behind hte scenes" detail?

@agardnerIT under the hood, scheduling gates are implemented as a PreEnqueue scheduler plugin. Our original scheduler is also a plugin.
As far as I know, there could be two interesting scenarios: (1) users have their own custom logic for scheduling implemented as a separate scheduler that has nothing to do with the default one and is not a plugin, (2) users also implemented a plugin to the default scheduler as we did in the previous versions of Keptn.

I do not think there will be many visible changes in either scenario: in case of (1) they need to choose between Keptn and their scheduler. Scheduling gates will not work since they modify the default scheduler behavior. With (2) scheduling gate should work out of the box, and then they could have used our scheduler plugin in combination with theirs so this worked before and works now. Only difference in case (2) was that before they had to tweek the scheduler plugin configuration to use both theirs and ours, now this is not needed anymore with k8s 1.27 and above.