-
Notifications
You must be signed in to change notification settings - Fork 39.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
service IPs and ports are not released when deleting a service via a finalizer-removing update #87603
Comments
/sig api-machinery |
Maybe also helpful: the output of kubectl events:
When these events get emmitted the problem does occur - so the repair seems to break things in this case. |
the service registry overrides the Delete implementation to free allocated ip/ports after a Delete API request: kubernetes/pkg/registry/core/service/storage/rest.go Lines 249 to 268 in 2cd90fb
but it does not implement an AfterDelete hook in its strategy, so it does not participate in object deletion in response to removing the last finalizer of a service with a deletionTimestamp set during an Update API request: kubernetes/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go Lines 553 to 573 in 2cd90fb
kubernetes/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go Lines 423 to 443 in 2cd90fb
kubernetes/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go Lines 1050 to 1057 in 2cd90fb
/sig network |
@liggitt: Those labels are not set on the issue: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/triage unresolved Comment 🤖 I am a bot run by vllry. 👩🔬 |
/remove-triage unresolved |
@MrHohn this one seems important. |
Thanks for the great analysis. My understanding is that we need to implement a AfterDelete hook for service - I will take a stab :) |
@MrHohn hello, any updates on this? |
@sparkoo Sorry for the delay, I implemented a fix locally and am working on a test at the moment. Looking to have a PR out this week. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
We need to revisit this soon |
I've seen this as well:
After deleting the chart from the system, I'm left with the controller service intact. 11m Normal EnsuringLoadBalancer service/nginx-ingress-ingress-nginx-controller Ensuring load balancer
11m Normal EnsuredLoadBalancer service/nginx-ingress-ingress-nginx-controller Ensured load balancer
3m19s Normal DeletingLoadBalancer service/nginx-ingress-ingress-nginx-controller Deleting load balancer
3m18s Warning FailedToCreateEndpoint endpoints/nginx-ingress-ingress-nginx-controller Failed to create endpoint for service default/nginx-ingress-ingress-nginx-controller: endpoints "nginx-ingress-ingress-nginx-controller" already exists
2m37s Warning PortNotAllocated service/nginx-ingress-ingress-nginx-controller Port 31861 is not allocated; repairing
2m37s Warning PortNotAllocated service/nginx-ingress-ingress-nginx-controller Port 32175 is not allocated; repairing
2m37s Warning ClusterIPNotAllocated service/nginx-ingress-ingress-nginx-controller Cluster IP 10.233.15.55 is not allocated; repairing |
Any updates on this?
|
I was impacted by this issue today, also got stuck trying to delete a service in a Kubernetes 1.18 cluster
|
For those who are only interested in deleting the service
|
This bug is a "temporary" problem when using finalizers on services, but doesn't cause the service to got stuck on deletion, if you have to remove the finalizer manually you should check what controller should remove that and what is causing it to fail to delete it |
heh, I manage to reproduce it #102955 |
/assign |
@aojea: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Thought I would add to this, Having the exact same issue with v1.18.16 on EKS. Initially, namespaces would be stuck in a "Terminating" state. Further research would lead you to find that services (loadbalancers) in Kubernetes were hanging or would persist for 10+ minutes. If you left it alone for a while, it would go away... eventually. However, this is typically instantly deleted. When running describe, I would notice it would initially say
Followed by the below about 60 seconds later:
Then the service would just sit there... until it got deleted ~10 minutes later. @lkoniecz solution of:
works perfect in the meantime, but going to have to overhaul a lot of automation. Going to downgrade to 1.17 as the issues seems to appear in any version above 1.17. |
What happened:
The Service "foo" is invalid: spec.ports[0].nodePort: Invalid value: 30003: provided port is already allocated
The apiserver logs the following lines prior to this happening:
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Example output:
Anything else we need to know?:
This does not always happen / this is a flakyness
Also the problem get's auto-resolved by the apiserver after some time (But this may need about 10 minutes):
E0128 08:01:24.562044 1 repair.go:300] the cluster IP 10.0.0.215 may have leaked: flagging for later clean up
Background for us here is: we want to run a custom controller for services of
type: LoadBalancer
and want to use a finalizer. We did hit this issue sometimes during dev.Environment:
kubectl version
):cat /etc/os-release
):uname -a
):Linux 5.3.0-26-generic #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
The text was updated successfully, but these errors were encountered: