Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use patch to set finalizers #317

Merged
merged 2 commits into from
Dec 6, 2019

Conversation

jackkleeman
Copy link
Contributor

@jackkleeman jackkleeman commented Dec 2, 2019

For pods in terminating state, their deletion timestamp is constantly
increasing, I'm not sure why but I suspect its do to with gatekeeper. As
a result, their resource version increases. This means that when we try
to remove the finalizer, there is an error 'this object has already
changed'. I think that using a patch will fix this.

{"level":"error","ts":1575303739.4736946,"logger":"controller","msg":"could not remove finalizer","kind":"Config","gvk":"/v1, Kind=Pod","name":"s-ledger-exporter-76877648cd-xmzwz","namespace":"io-gmon","error":"Operation cannot be fulfilled on pods \"s-ledger-exporter-76877648cd-xmzwz\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/open-policy-agent/gatekeeper/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/go-logr/zapr/zapr.go:128\ngit.luolix.top/open-policy-agent/gatekeeper/pkg/controller/config.(*finalizerCleanup).Clean.func1\n\t/home/travis/gopath/src/github.com/open-policy-agent/gatekeeper/pkg/controller/config/config_controller.go:347\ngit.luolix.top/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/home/travis/gopath/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203\ngit.luolix.top/open-policy-agent/gatekeeper/pkg/controller/config.(*finalizerCleanup).Clean\n\t/home/travis/gopath/src/github.com/open-policy-agent/gatekeeper/pkg/controller/config/config_controller.go:377"}

@maxsmythe
Copy link
Contributor

No objections to changing this to patch.

I'm curious why you think gatekeeper would change the deletion timestamp? Gatekeeper does not modify those.

I do see this bit of information WRT deletion timestamps and pods, unsure if it's relevant:

kubernetes/kubernetes#71494

@jackkleeman
Copy link
Contributor Author

jackkleeman commented Dec 2, 2019

I'm curious why you think gatekeeper would change the deletion timestamp

Just a guess. Something pretty strange is going on!

@jackkleeman
Copy link
Contributor Author

jackkleeman commented Dec 2, 2019

@maxsmythe are there integration tests which explicitly test for finalizer removal?

@maxsmythe
Copy link
Contributor

Yep, the config controller tests look for sync finalizer removal:

return errors.New("testns namespace still has sync finalizer")

@ritazh
Copy link
Member

ritazh commented Dec 3, 2019

Would be nice to add a test for sync finalizer removal in e2e, similar to here: https://github.com/open-policy-agent/gatekeeper/blob/master/test/bats/test.bats#L82

@maxsmythe
Copy link
Contributor

@ritazh Is this blocking for you?

@ritazh
Copy link
Member

ritazh commented Dec 5, 2019

no. I have created #323 to track.

For pods in terminating state, their deletion timestamp is constantly
increasing, I'm not sure why but I suspect its do to with gatekeeper. As
a result, their resource version increases. This means that when we try
to remove the finalizer, there is an error 'this object has already
changed'. I think that using a patch will fix this.

Signed-off-by: Jack Kleeman <jackkleeman@gmail.com>
@maxsmythe maxsmythe merged commit 154da92 into open-policy-agent:master Dec 6, 2019
@jackkleeman jackkleeman deleted the patch-finalizers branch December 6, 2019 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants