Research :: Readiness Scheduling Gates

[thisthat]

Goal

Prototype a KLT implementation that uses Pod Scheduling Readiness instead of the Scheduling Plugin.

Technical Details

KLT currently uses a custom plugin for the scheduler to forbid Pods from being bound to a node. With the new addition of the Readiness Gates, we could eliminate the extra binary and use the new API. This has some consequences for our mutating webhook: it should add a gate when a manifest is applied (on update it won't). When the pre-checks are completed successfully, the gates shall be removed.

Resources

• Changelog: https://kubernetes.io/blog/2023/04/11/kubernetes-v1-27-release/#pod-scheduling-readiness-goes-to-beta

[thisthat]

Old PoC that used a workaround with extra labels: #632

[thisthat]

PR kubernetes-sigs/controller-runtime#2189 has been merged and unlocks this research

[RealAnna]

The new controller runtime v 0.15.0 will introduce multiple breaking changes that will require code refactoring/changes in all our operators:

• the server struct for the webhooks is now an interface, so we need to change all reference of it since we use it to override configs when setting up certificates
• the decoder injector has been removed this means we need to explicitly pass one to the webhooks, as written in Remove Decoder Injector interface from webhook #1445
• few ways to setup the controller manager are deprecated: to set up the port we should use WebhookServer.Port instead and for ClientDisableCacheFor weshould use Client.Cache.DisableCacheFor.
• multiple controller unit tests still fail in the poc after the new library version, so they need refactoring. Maybe I missed even other breaking changes 😿

After these changes and the import of controller runtime and k8s 1.27.1 libraries, we can substitute the scheduler with gates, in the poc this is done by adding an extra webhook reacting to the creation of new pods:

• extract common functions from the mutating webhook
• introduce a gating webhook that simply adds a keptn gated at creation
• remove the gate in the webhook instance controller after all pre check passed
• modify operator/config/default/webhooknamespaces_patch.yaml and operator/config/webhook/manifests.yaml to include the new webhook
• add the extra webhook in the list passed to builder.run in operator/main.go
• create new test and adapt existing ones, this is partially done in the poc poc: scheduling gates beta verison #1249

It is left do decide also what to do with the scheduler. Shall we do a breaking change and delete it or add some logic to enable/disable gates or scheduler for at least one version?