kustomize-controller: Use cli-utils apply package

[seaneagan]

The cli-utils apply package which is used by kpt live apply has a bunch of useful functionality, some of which overlaps with kustomize-controller functionality, e.g. pruning. It uses an inventory object instead of labels to determine package membership. This has the advantage of not needing to modify the payload objects themselves. The inventory object id could be added to the Kustomization status. This could be a nice UX for determining what resources the last kustomization reconciliation produced. It could also allow for integration with e.g. kpt live status (beyond the existing kstatus health checks, as for example we can't include all status output in the CR status conditions).

[stefanprodan]

We label objects so that users can trace back what things on the cluster originated from which Kustomization. As for pruning, the cli utils doesn’t handle objects stuck in finlization, maybe for kpt which is a CLI is not a problem but for kustomize controller this means it can no longer reconcile new revisions, we have this problem with Flux v1 and it’s a major pain.

[stefanprodan]

I think something similar could be accomplished using an inventory object

Users shoudn't have to deal with an inventory object, right now they can do it with just kubectl:

$ kubectl get deployments \
-l=kustomize.toolkit.fluxcd.io/name=test,kustomize.toolkit.fluxcd.io/namespace=gotk-system \
--all-namespaces 

The pruning has some other niceties like calculating an appropriate ordering of the deletions

Kustomize controller does its own calculation such that CRs can be finalized before deleting the CRDs.

What issues have you encountered in kustomize-controller's GC that would be solved with cli-utils?

[seaneagan]

Users shoudn't have to deal with an inventory object, right now they can do it with just kubectl:

$ kubectl get deployments \
-l=kustomize.toolkit.fluxcd.io/name=test,kustomize.toolkit.fluxcd.io/namespace=gotk-system \
--all-namespaces 

I agree the labels are definitely useful for that. The inventory object in cli-utils is specifically about the pruning use case.

What issues have you encountered in kustomize-controller's GC that would be solved with cli-utils?

None at all, just wanted to look into the possibility of aligning on a common library with similar functionality, but sounds like it may not satisfy the needs of kustomize-controller. Thanks for the responses!

[stefanprodan]

I'm for reusing kustomize libraries as much as we can, but GC is a super sensitive area that I think we should approach from a controller perspective rather than a CLI/interactive way. Looking ahead, we will probably need to extend our GC to treat CAPI objects differently, using some library that we don't control will make it super hard to deal with edge cases.

[seaneagan]

@stefanprodan #324 is another use case for the inventory object. What do you think about using cli-utils (for the inventory object) without using its pruning functionality?

[stefanprodan]

Why would we need the the inventory object for that? The Kustomize GC snapshot has all the data for selecting the objects by kind/label and feed them to the kstatus.