[RFC] Server-side reconciliation for the v1beta2 API

[stefanprodan]

This PR introduces a new reconciler based on Kubernetes server-side apply and graduates the API to v1beta2. Part of fluxcd/flux2#1889.

💡 Motivation

• Improve performance (CPU, memory, network, FD usage) and reduce the number of calls to Kubernetes API by replacing kubectl execs with a specialized applier written in Go.
• Being able to validate and reconcile sources that contain both CRDs and CRs.
• Detect and report drift between the desired state (git, s3, etc) and cluster state reliably.
• Preview local changes to manifests without pushing to upstream (flux diff -k command TBA).
• Being able to wait for all applied resources to become ready without requiring users to fill in the health checks list.
• Improve the overall observably of the reconciliation process by reporting in real-time the garbage collection and health assessment actions.

⏳ Background

When we started Flux v2, we've set a goal to stop relying on 3rd party binaries for core features. While we've successfully replaced the Git CLI shell execs with Go libraries (go-git, git2go) and C libraries (libgit2, libssh2), the kustomize CLI with Go libraries (kustomize/api, kustomize/kyaml), we're still depending on the kubectl CLI for the 3-way-merge apply feature. With Kubernetes server-side apply being promoted to GA, we can finally get rid of kubectl and drive the reconciliation using exclusively the controller-runtime Go client.

💥 Breaking changes

• Namespaced objects must contain metadata.namespace, defaulting to the default namespace is no longer supported.
• The logs, events and alerts that report Kubernetes namespaced object changes are now using the Kind/Namespace/Name format instead of Kind/Name e.g.:

Service/flux-demo/podinfo unchanged
Deployment/flux-demo/podinfo configured
HorizontalPodAutoscaler/flux-demo/podinfo deleted

• The minimum supported version of Kubernetes has changed to:

Kubernetes version	Minimum required
v1.16	>= 1.16.11
v1.17	>= 1.17.7
v1.18	>= 1.18.4
v1.19 and later	>= 1.19.0

All the versions above fix a regression in the managed fields and field type.

API changes

The kustomize.toolkit.fluxcd.io/v1beta2 API is backwards compatible with v1beta1.

Additions and deprecations:

• .spec.validation deprecated (server-side validation is implicit)
• .spec.patchesStrategicMerge deprecated in favour of .spec.patches
• .spec.patchesJson6902 deprecated in favour of .spec.patches
• .status.snapshot replaced by .status.inventory
• .spec.wait added (when enabled the controller will wait for all the reconciled resources to become ready)

Reconciler changes

The server-side reconciler comes with the following behavioural changes:

• Creates an inventory of objects to be applied (persisted in-cluster under .status.inventory).
• Applies first custom resource definitions (CRDs) and namespaces, waits for them to register and only then applies the custom resources.
  Fixes: Cope when syncing a CRD and resources using the CRD flux2#1425
• Validates all resources with server-side dry-run apply (includes validation webhooks).
• Reconciles only the resources that are in drift (viewing the diff could be made available in flux CLI or via trace events).
  Fixes Detect changes/drift through a filtered diff of the apply/prune #352
  Ref Dry-run support #213
• Handles apply timeouts gracefully and reports the involved resource kind, namespace and name (kubectl is no longer used for any operation).
  Fixes When webhooks block kubectl apply and the command times-out, flux should log it as a timeout instead of an empty error #311
• Prunes the objects that were previously applied but are missing from the current inventory (the garbage collector no longer annotates the objects after each reconciliation).
  Fixes Kustomization: garbage collection website#498
  Ref Garbage Collection deleting Endpoint resources created by Service resource #415
• Emits change events based on the dry-run diff, and only for the resources that where created, configured or deleted.
  Fixes Kustomize applies and notifies when no changes occur #413
  Fixes The log shows that the resource is configured, but in fact it is not so #403
  Fixes Notifications - information content and filtering of info alerts flux2#710
  Fixes Slack notification is misleading flux2#626
  Supersedes Detect changes/drift through a filtered diff of the apply #379
• Reports the health assessment progress and its timeout. The health check timeout defaults to 30sec less than the reconciliation interval. The minimum timeout is set at 30sec.
  Fixes Writing an invalid health assessment seems to break kustomize controller #405
• Emits health check events only when the previous status changed.
  Fixes Incorrect state in health check event #191
• When .spec.wait is true, the controller runs the health assessment for all resources ignoring the ones in spec.healthChecks.
  Fixes Health assessment for all resources of a given kind. #197
  Ref kustomize-controller: ability to automatically health check all resources flux2#324
• Finalize objects under abnormal conditions. The garbage collector no longer blocks the finalization, instead it issues an event with the staled resources.
  Fixes Reconciler error - Unable to prune for finalizer - failed to build kube client for Kustomization: ServiceAccount not found flux2#997
• Reconcile empty sources including pruning all the resources previously applied.
  Fixes kustomization.yaml with no resources fails #391
  Fixes Garbage collection step is skipped if there are no manifests in the repo #187
• All the operations involving Kubernetes Secrets produce logs and events with the secret data masked.
  Fixes Notification controller secrets leakage on patch operation flux2#1887
• Add support for YAML anchors (kustomize v4.4.0).
  Ref Breaking changes in Flux due to Kustomize v4 flux2#1522

Testing

The SSA reconciler and its controller are tested using Go stdlib, gomega and controller-runtime envtest:

 github.com/fluxcd/kustomize-controller/controllers coverage:  71.2% of statements

Due to the controller-runtime envtest limitations (no kube-controller-manager), Kubernetes Kind e2e tests where added to the GitHub Actions workflow to cover features such as waiting for deployments rollout and CRDs+CRs staged apply.

Feedback

Please comment on this PR and let us know your thoughts about this.

If you want to try out the v1beta2 API on your own test cluster:

1. Install the latest Flux controllers

flux install

2. Apply the CRDs from this branch

 kubectl apply -k https://github.com/fluxcd/kustomize-controller/config/crd?ref=v1beta2

3. Deploy the kustomize-controller build of this branch

kubectl -n flux-system set image deployment/kustomize-controller \
manager=ghcr.io/fluxcd/kustomize-controller:v1beta2-50c71354

What's next?

• Move the SSA resource manager to fluxcd/pkg/ssa
• Use the SSA manager in Flux CLI to replace kubectl shell execs for flux bootstrap and flux install
• Bump the minimum Kubernetes version to 1.18.8 in flux check --pre

[hiddeco]

We have had a discussion about this in the past, but I think it would be wise if we would make the finalizers domain specific. Reason for this is that the pattern of using various finalizers to control the garbage collection process of an object if multiple reconcilers are dealing with an object has become more common.

I think your objection in the past was that removing "all Flux reconcilers" in cases where they get stuck would become more difficult, but if we would nest all under a sub(domain)name, you could still loop over all that match that suffix. E.g. applier.finalizers.fluxcd.io, source.finalizers.fluxcd.io.

[stefanprodan]

If we change this now, wouldn't result in orphan resources after a Flux upgrade? I would introduce this change into a followup PR.

[hiddeco]

Follow up PR (e.g. bundled with the other refactoring efforts) is fine with me, just wanted to have it noted given it showed up again :-)

[somtochiama]

Tested this PR on kubernetes version 1.16..13, 1.17.17, 1.18.8, 1.19.1, 1.20.2 and it works💥.
It fails on 1.18.2 with the following error:

{"level":"error","ts":"2021-09-13T15:02:33.963Z","logger":"controller.kustomization","msg":"unable to update status after reconciliation","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"podinfo","namespace":"flux-system","error":"failed to update object (json PATCH for kustomize.toolkit.fluxcd.io/v1beta2, Kind=Kustomization) managed fields: failed to update ManagedFields: failed to convert new object: .status.inventory: field not declared in schema"}
{"level":"error","ts":"2021-09-13T15:02:33.964Z","logger":"controller.kustomization","msg":"Reconciler error","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"podinfo","namespace":"flux-system","error":"failed to update object (json PATCH for kustomize.toolkit.fluxcd.io/v1beta2, Kind=Kustomization) managed fields: failed to update ManagedFields: failed to convert new object: .status.inventory: field not declared in schema"}
{"level":"info","ts":"2021-09-13T15:02:34.495Z","logger":"controller.kustomization","msg":"server-side apply completed","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"podinfo","namespace":"flux-system","output":{"Deployment/flux-system/podinfo":"unchanged","HorizontalPodAutoscaler/flux-system/podinfo":"unchanged","Service/flux-system/podinfo":"unchanged"}}

This is likely due to k/k#91748

[squaremo]

The kustomize.toolkit.fluxcd.io/v1beta2 API is backwards compatible with v1beta1.
[...]

• .spec.validation removed (server-side validation is implicit)

Is that field ignored or removed, and if the latter, does this mean a v1beta1 resource won't validate as v1beta2?

[stefanprodan]

Is that field ignored or removed, and if the latter, does this mean a v1beta1 resource won't validate as v1beta2?

I haven't bumped the API version in the e2e tests to prove that the v1beta1 CRs work as expected. What happens is that the Kubernetes API drops all unknown fields before validating the CR, so validation works since .spec.validation is removed.

[squaremo]

• When .spec.wait is true, the controller runs the health assessment for all resources ignoring the ones in spec.healthChecks.
  Fixes Health assessment for all resources of a given kind. #197

In the linked issue, it's explained that waiting on everything wouldn't work because some resource types are incompatible with kstatus (and this is the motivation for suggesting an allow-list or patterns). The change quoted above would make the controller wait on everything -- so is it the case that

1. the issue report is mistaken, and it's fine to wait on everything
2. the mechanism here works around incompatibility with kstatus
3. this will not work when you're using such resources

[stefanprodan]

@squaremo all Kubernetes builtin kinds are compatible with kstatus. As for custom resources, if the those have no status, or no ready condition in status, or no observed generation in status, then kstatus instead of crashing will report those resources as ready.

[hiddeco]

Although the PR seems to be immense in LOC, most of the changes turned out to actually be tests.

All looks sensible to me @stefanprodan, nice work 🥇💯

[dsogari]

Why can't we have the option to disable the dry-run behavior? The implicit behavior fails for me when deploying a kube-prometheus stack:

Pod/monitoring/kube-prometheus-grafana-test dry-run failed, reason: Forbidden, error: pods "kube-prometheus-grafana-test" is forbidden: error looking up service account monitoring/kube-prometheus-grafana-test: serviceaccount "kube-prometheus-grafana-test" not found

It fails because the service account mentioned is only created during the actual reconciliation.

[kingdonb]

@dsogari Please take your inquiry to a new issue, or https://github.com/fluxcd/flux2/discussions

It is not likely to get positive attention here (necro-bumped a PR that is several months old), and there is not enough information in your post to help you, nor this is not the forum.