Ignore manifests using annotation

[raffis]

One feature I miss from flux is annotating resources on k8s directly using fluxcd.io/ignore to ignore them temporarily.
This seems not possible in gotk, you need to suspend the ks pipeline which also stops syncing the entire source and not just a single manifest.

I used fluxcd.io/ignore quite often to debug things.

Also in gotk an example where the suspend & debug does not work: If you have the flux components in the same ks pipeline as other infra things and want to change for example the log level to debug of the ks controller it won't be possible since it gets overwritten again but you can't suspend since you want to debug the ks controller itself. One would need to change this in the source which is not a nice workflow for debugging.

Can we add an annotation to ignore resources? This would help a lot to debug things.

[stefanprodan]

Please see the docs: https://toolkit.fluxcd.io/components/source/gitrepositories/#excluding-files

[raffis]

Please see the docs: https://toolkit.fluxcd.io/components/source/gitrepositories/#excluding-files

This works thanks for the hint.

However the downside of this approach is you need to know the source. You can't ignore things from a k8s perspective.
So this does not fully replace the annotation.
On the other hand you can exclude multiple things at once and also see all ignores.

[stefanprodan]

However the downside of this approach is you need to know the source. You can't ignore things from a k8s perspective.

Yes, using the cluster to ignore manifests was an unfortunate choice in Flux1 as it has many downsides, like querying the Kubernetes API before an apply. This is not going to be implemented in v2.

[racdev]

Can I ask how excluding files in a gitrepository solves the use case here as I can't get my head around how that would work?

From reading those Doc's it sounds like excluding certain files means they aren't included in the set of pulled files that the kustomize controller then applies. If that is the case then if I for example want to ignore a deployment defined in a particular yaml file (so that I can manually change it to test a small change for example), and I add that file to the ignore list, won't the kustomize then complain that the referenced file doesn't exist? Or is it that you would ignore the entire set of files that are bundled as part of that deployment (the kustomization.yaml etc)? Finally, if these files are ignored and not available to the kustomize controller won't it then garbage collect the deployment as it is no longer referenced?

We are currently migrating from flux 1 to 2 and use the flux ignore annotation extensively so it would be good to understand how we can change our current approach to work with flux 2.

[stefanprodan]

In v2 you can suspend the reconciliation of a Kustomization, this feature can be used when you need to manually edit Kubernetes objects, once you are done editing, you can resume the reconciliation.

[racdev]

Yeah I understand I can suspend the kustomization, the only problem with that (other than I would need to give developers the additional rbac to do that manually) is that we would in effect have to suspend all reconciliation in our cluster as the kustomize that applies application deployments is managed by a core cluster kustomize that applies our core deployments as well as the kustomize for applications. This is because in flux 1 we minimised the number of flux instances by having a core instance that then manages an apps instance (as well as core deployments)

I suppose what we need to do is split all our deployments out into their own separate kustomize reconciliations and to have the core kustomization only manage flux2 resources - we can then suspend the core kustomization and any app specific one to test changes manually, without affecting other reconciliations. I assume that that is the recommended approach and setup for using flux2?

[raffis]

Yes I would definitely split them at least that is what we're going to do. One ks for infra and more for different env/teams or apps.

@stefanprodan One thing regarding the annotation. Can we add a safety check to the ks controller to ignore the manifests from the source which have that annotation? We got surprised by one deployment being back and realized it was checked in in the git repo with that annotation.

[stefanprodan]

I suppose what we need to do is split all our deployments out into their own separate kustomize reconciliations and to have the core kustomization only manage flux2 resources - we can then suspend the core kustomization and any app specific one to test changes manually, without affecting other reconciliations. I assume that that is the recommended approach and setup for using flux2?

Yes

We got surprised by one deployment being back and realized it was checked in in the git repo with that annotation.

There is no annotation, suspend/resume works by toggling spec.suspend. You should add a validation step in CI that rejects PRs that are setting this field in git.

[raffis]

There is no annotation, suspend/resume works by toggling spec.suspend. You should add a validation step in CI that rejects PRs that are setting this field in git.

Yes absolutely but this happened for an existing git repo which we took over. Yes I know the safer way would have been to migrate the entire repo. But that might still have happened because people might not be aware of this. If you don't add the check in the code at least a warning in the migration docs would be a good idea.

[racdev]

I would second the need for the docs to be updated with regards to the flux ignore flag as it may catch people out. It is not mentioned anywhere in the flux 2 docs, either as an entry in the migration docs or in the FAQ, but had a specific entry in the flux 1 FAQ. I only realised it wasn't supported due to this issue being raised here.

[stefanprodan]

@racdev this is good feedback, thank you! Can you please open an issue on flux2 repo for the FAQ?

[racdev]

fluxcd/flux2#614 created

[ringerc]

Anyone finding this with Google who's looking for the fluxcd v2 spelling, use kustomize.toolkit.fluxcd.io/reconcile: disabled

See fluxcd/website#623 and the flux v2 FAQ