Slack notification is misleading

[Marx2]

Despite changing only one yaml in GIT, Slack notification from Slack lists all yaml as "configured".
I expect it to send notifictaion only about yamls which has been changed

[stefanprodan]

There is no way around this, you can disable pruning so that objects are no longer labeled with the git revision. Garbage collection requires that all objects have a label to keep track of manifests removed from git, so the message is not misleading, it tells you what actually happened.

[Marx2]

I'm not sure how Flux works under the hood, but if I try to apply yaml, which doesn't change, I have infromation from kubernetes, that there was no change. Also kubcetl diff can detect if there is difference. So I don't understand, why Flux cannot detect only yamls that changed, and display this info. As for now all notifications looks the same and gives no value besides "something has changed". Maybe you remember sha commits, I'm not as good ;)

[hiddeco]

Try applying the same directory with kubectl apply -k or kubectl apply -f and see what the output is :-)

[Marx2]

I do this daily. Output is no change or changed, depending on fact, if yaml was different or not pt., 18 gru 2020, 19:25 użytkownik Hidde Beydals <notifications@github.com> napisał:

…

Try applying the same directory with kubectl apply -k or kubectl apply -f and see what the output is :-) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#626 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGI4ZRCUKWDJXW5NDXGQ4TSVONAXANCNFSM4VBCIGDQ> .

[stefanprodan]

If you want Flux to apply without deleting objects, then set prune to false and it will no longer change the labels for each revision.

[Marx2]

Ok, i see those labels on deployments and now understand why all deployments are changed, hence the message. Is there any description how to define "prune" in Flux and what are consequences?

Is it "prune: true" in gotk-sync.yaml?

[stefanprodan]

docs here https://toolkit.fluxcd.io/components/kustomize/kustomization/#garbage-collection

[jonaskello]

I also have this issue and found that when the alert always says everything is changed the information of what has changed is not really useful anymore. I think the ideal solution would be if flux could alert what objects changed, disregarding objects that only has GC label changes.

I appreciate this would be hard to do and of course this would not be possible by just alerting the output of kustomize which I guess it is doing now? I'll make some naïve suggestions since I don't know all details of how flux works:

1. Flux could do something similar to kubectl -k diff before adding new GC labels and keep track of those objects because they are probably the ones that will be changed for other reasons than GC labels? I guess this is not ideal since the alert would then not reflect what kustomize said happened. It will be more of a guess what should have happened.

2. Flux runs kustomize with the changes from git without applying any GC labels. It then does a second run to add the GC labels. Only the first run is alerted. This is not ideal becuase the kustomize run is not atomic but probably better than (1).

[arsiesys]

In case of someone find this case while digging a similar issue. We have many deployment that always get marked as "configured" and then get included in the alerts. It seems to be related to a kubernetes behavior that sometimes mark an "kubectl apply -f" as configured while the "kubectl diff -f" would return no change. The kubectl client sometimes send a PATCH that change nothing (it's not clear why yet for me..) and that trigger the answer "configured" used by flux to determine if the file was changed or not.

So, obviously we can't really trust kubectl for the "configured" output :(.

kubernetes/kubernetes#66450