upstream IP inconsistent with pod IP after pod deletion #768

caseylucas · 2017-05-26T00:17:10Z

Problem

We noticed that after pod deletion and waiting for pods to restart, we were getting errors (500, 502, etc.) Once the errors start for a virtual host, they remain until we restart the ingress.

Log

In nginx log, we noticed connection refused.

2017/05/25 22:50:47 [error] 2251#2251: *34997 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: uloyl-srv.XXX, request: "OPTIONS /api/v1/activity? HTTP/1.1", upstream: "http://10.2.1.25:5000/api/v1/activity?", host: "uloyl-srv.XXX"

Note the upstream IP: 10.2.1.25
I assume that 10.2.1.25 is the previous pod's IP.
The pod's current IP (after pod deletion and auto restart) is actually: 10.2.3.14

Within the cluster I can use curl to hit the pod's ip (10.2.3.14) and get back expected results.

Versions

ingress:

Image:        gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.6
Args:
      /nginx-ingress-controller
      --default-backend-service=$(POD_NAMESPACE)/nginx-ingress-default-http-backend
      --default-ssl-certificate=$(POD_NAMESPACE)/wildcard-cert
      --configmap=$(POD_NAMESPACE)/nginx-load-balancer-conf
      --v=3

k8s: v1.4.5

Pod Info

kubectl get pods | grep loyl-backend
loyl-backend-3107245711-hmfab                     1/1       Running            0          56m

kubectl describe pod loyl-backend-3107245711-hmfab
Name:        loyl-backend-3107245711-hmfab
...
Labels:      app=loyl-backend
             pod-template-hash=3107245711
Status:      Running
IP:          10.2.3.14
...

nginx.conf oddness

kubectl exec nginx-ingress-lb-2j1ch cat /etc/nginx/nginx.conf > nginx.conf.1
...
    upstream default-loyl-backend-service-5000 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 10.2.1.25:5000 max_fails=0 fail_timeout=0;
    }
...

Note the same IP as in the connection refused log message.

kubectl delete pods nginx-ingress-lb-2j1ch
pod "nginx-ingress-lb-2j1ch" deleted

Dump new nginx.conf:

kubectl exec nginx-ingress-lb-3zv7l cat /etc/nginx/nginx.conf > nginx.conf.2

I was killing random pods during testing. Others look wrong but the last one on port 5000 is the one I originally noticed was wrong.

diff nginx.conf.1 nginx.conf.2
179c179
<         server 10.2.0.25:8088 max_fails=0 fail_timeout=0;
---
>         server 10.2.0.21:8088 max_fails=0 fail_timeout=0;
184c184
<         server 10.2.2.19:3000 max_fails=0 fail_timeout=0;
---
>         server 10.2.0.24:3000 max_fails=0 fail_timeout=0;
189c189
<         server 10.2.0.24:8090 max_fails=0 fail_timeout=0;
---
>         server 10.2.2.20:8090 max_fails=0 fail_timeout=0;
199c199
<         server 10.2.1.25:5000 max_fails=0 fail_timeout=0;
---
>         server 10.2.3.14:5000 max_fails=0 fail_timeout=0;
----

After restart, all is good because the configs are correct again.

The text was updated successfully, but these errors were encountered:

caseylucas · 2017-05-28T17:00:16Z

Any pointers on how to best debug this problem would be appreciated as I'm not very familiar with
the code base. I cannot easily reproduce this issue but instead kill pods until I see the
problem occurr. This is what I have so far...

I added in a debug message showing the old and cur parameters to the UpdateFunc
(https://github.com/kubernetes/ingress/blob/master/core/pkg/ingress/controller/controller.go#L230)
After killing pods, the data seems to be correct. The old pod IP goes away and the new pod IP comes
in. However the nginx.conf file is not updated. Ex logs:

I0528 15:43:02.766065       5 controller.go:239] in UpdateFunc: not DeepEqual: old: &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:loyl-backend-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/loyl-backend-service,UID:43f2ca4a-1356-11e7-8a38-0ae5698449a8,ResourceVersion:15201213,Generation:0,CreationTimestamp:2017-03-28 01:31:28 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: loyl-backend-service,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.2.2.17  0xc4209c5e50 ObjectReference{Kind:Pod,Namespace:default,Name:loyl-backend-3347238716-c4e1f,UID:b41aef89-4348-11e7-8343-0ae5698449a8,APIVersion:,ResourceVersion:15201211,FieldPath:,}}] [] [{http 5000 TCP}]}],}, cur &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:loyl-backend-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/loyl-backend-service,UID:43f2ca4a-1356-11e7-8a38-0ae5698449a8,ResourceVersion:15313162,Generation:0,CreationTimestamp:2017-03-28 01:31:28 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: loyl-backend-service,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0528 15:43:17.362050       5 controller.go:239] in UpdateFunc: not DeepEqual: old: &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:loyl-backend-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/loyl-backend-service,UID:43f2ca4a-1356-11e7-8a38-0ae5698449a8,ResourceVersion:15313162,Generation:0,CreationTimestamp:2017-03-28 01:31:28 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: loyl-backend-service,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}, cur &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:loyl-backend-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/loyl-backend-service,UID:43f2ca4a-1356-11e7-8a38-0ae5698449a8,ResourceVersion:15313389,Generation:0,CreationTimestamp:2017-03-28 01:31:28 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: loyl-backend-service,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.2.0.26  0xc42043a840 ObjectReference{Kind:Pod,Namespace:default,Name:loyl-backend-3347238716-52m6w,UID:559ec612-43bc-11e7-8343-0ae5698449a8,APIVersion:,ResourceVersion:15313386,FieldPath:,}}] [] [{http 5000 TCP}]}],}

There was no message in the log showing the diffed nginx.conf. New pod IP should be 10.2.0.26 but is the original:

kubectl exec nginx-ingress-lb-p17zx cat /etc/nginx/nginx.conf | grep -A 4 "upstream default-loyl-back"
    upstream default-loyl-backend-service-5000 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 10.2.2.17:5000 max_fails=0 fail_timeout=0;
    }

I also verified the new pod IP from k8s perspective:

kubectl get pods | grep loyl-back | awk '{print $1}' | xargs kubectl describe pods | grep IP
IP:        10.2.0.26

Once the controller gets into this bad state no more nginx.conf updates occur. Log messages showing the UpdateFunc call after another pod deletion:

I0528 16:20:58.625565       5 controller.go:239] in UpdateFunc: not DeepEqual: old: &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:loyl-backend-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/loyl-backend-service,UID:43f2ca4a-1356-11e7-8a38-0ae5698449a8,ResourceVersion:15313389,Generation:0,CreationTimestamp:2017-03-28 01:31:28 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: loyl-backend-service,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.2.0.26  0xc42043a840 ObjectReference{Kind:Pod,Namespace:default,Name:loyl-backend-3347238716-52m6w,UID:559ec612-43bc-11e7-8343-0ae5698449a8,APIVersion:,ResourceVersion:15313386,FieldPath:,}}] [] [{http 5000 TCP}]}],}, cur &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:loyl-backend-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/loyl-backend-service,UID:43f2ca4a-1356-11e7-8a38-0ae5698449a8,ResourceVersion:15318560,Generation:0,CreationTimestamp:2017-03-28 01:31:28 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: loyl-backend-service,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}
I0528 16:20:59.675812       5 controller.go:239] in UpdateFunc: not DeepEqual: old: &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:loyl-backend-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/loyl-backend-service,UID:43f2ca4a-1356-11e7-8a38-0ae5698449a8,ResourceVersion:15318560,Generation:0,CreationTimestamp:2017-03-28 01:31:28 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: loyl-backend-service,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[],}, cur &Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:loyl-backend-service,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/endpoints/loyl-backend-service,UID:43f2ca4a-1356-11e7-8a38-0ae5698449a8,ResourceVersion:15318585,Generation:0,CreationTimestamp:2017-03-28 01:31:28 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{app: loyl-backend-service,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Subsets:[{[{10.2.3.20  0xc420442420 ObjectReference{Kind:Pod,Namespace:default,Name:loyl-backend-3347238716-g7pl4,UID:a22694f2-43c1-11e7-8343-0ae5698449a8,APIVersion:,ResourceVersion:15318582,FieldPath:,}}] [] [{http 5000 TCP}]}],}

There is no nginx.conf diff update message in the log and the IP is still the original:

kubectl exec nginx-ingress-lb-p17zx cat /etc/nginx/nginx.conf | grep -A 4 "upstream default-loyl-back"
    upstream default-loyl-backend-service-5000 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 10.2.2.17:5000 max_fails=0 fail_timeout=0;
    }

I even tried deleting and adding new ingress definitions. The nginx.conf file gets no updates whatsoever.

Any tips on digging deeper?

aledbf · 2017-05-28T17:08:19Z

@caseylucas I cannot reproduce this issue with k8s 1.6 or 1.5.7

caseylucas · 2017-05-28T17:32:09Z

@aledbf So you think it's related to 1.4? Indeed, it is tough to replicate for me too however I'd like to avoid having to upgrade our k8s cluster (for now) if possible. Maybe some recommended spots to dig deeper?

caseylucas · 2017-05-29T15:52:39Z

@aledbf I was able to get get a trace after things seemed to lock up. Keep in mind I'm not a golang expert but I think that the problem may be an attempted recursive write lock of a RWMutex. I added a few debug log statements so the line number may be slightly off but you should be able to follow the stack trace. See the two lines "// CASEY: " below.

At k8s.io/client-go/tools/cache/delta_fifo.go:451, f.lock is held and process is called. controller.(*GenericController).controllersInSync is eventually called which calls back into delta_fifo attempting to acquire f.lock again in cache.(*DeltaFIFO).HasSynced

goroutine 44 [semacquire, 539 minutes]:
sync.runtime_SemacquireMutex(0xc42012e3c4)
  /usr/local/Cellar/go/1.8.1/libexec/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc42012e3c0)
  /usr/local/Cellar/go/1.8.1/libexec/src/sync/mutex.go:87 +0x9d
sync.(*RWMutex).Lock(0xc42012e3c0)
  /usr/local/Cellar/go/1.8.1/libexec/src/sync/rwmutex.go:86 +0x2d
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).HasSynced(0xc42012e3c0, 0x1dcd400) // CASEY: attempted recursive lock of DeltaFIFO.lock
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:167 +0x36
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*controller).HasSynced(0xc42020de80, 0x1)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/controller.go:126 +0x33
k8s.io/ingress/core/pkg/ingress/controller.(*GenericController).controllersInSync(0xc4201e2640, 0x154c846)
  /Users/clucas/go/src/k8s.io/ingress/core/pkg/ingress/controller/controller.go:334 +0xc0
k8s.io/ingress/core/pkg/ingress/controller.(*GenericController).syncSecret(0xc4201e2640)
  /Users/clucas/go/src/k8s.io/ingress/core/pkg/ingress/controller/backend_ssl.go:40 +0x89
k8s.io/ingress/core/pkg/ingress/controller.newIngressController.func4(0x1514d60, 0xc4203610f0, 0x1514d60, 0xc4204503c0)
  /Users/clucas/go/src/k8s.io/ingress/core/pkg/ingress/controller/controller.go:207 +0x73
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(0x0, 0xc420241d90, 0xc420241da0, 0x1514d60, 0xc4203610f0, 0x1514d60, 0xc4204503c0)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/controller.go:199 +0x5d
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate(0xc4204423e0, 0x1514d60, 0xc4203610f0, 0x1514d60, 0xc4204503c0)
  <autogenerated>:55 +0x87
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.NewInformer.func1(0x13bbfa0, 0xc4209f0380, 0x13bbfa0, 0xc4209f0380)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/controller.go:265 +0x32c
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc42012e3c0, 0xc4202b5f50, 0x0, 0x0, 0x0, 0x0) // CASEY: DeltaFIFO.lock is locked here
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 +0x27e
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop(0xc42020de80)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/controller.go:147 +0x40
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*controller).(k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x2a
k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4205e5fb0)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x5e
k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc42090bfb0, 0x3b9aca00, 0x0, 0x1315901, 0xc42046d2c0)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0xbd
k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4205e5fb0, 0x3b9aca00, 0xc42046d2c0)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*controller).Run(0xc42020de80, 0xc42046d2c0)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x237
created by k8s.io/ingress/core/pkg/ingress/controller.GenericController.Start
  /Users/clucas/go/src/k8s.io/ingress/core/pkg/ingress/controller/controller.go:1197 +0x1aa

There are other goroutines blocked in HasSynced. One more shown here:

goroutine 47 [semacquire, 539 minutes]:
sync.runtime_SemacquireMutex(0xc42012e3c4)
  /usr/local/Cellar/go/1.8.1/libexec/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc42012e3c0)
  /usr/local/Cellar/go/1.8.1/libexec/src/sync/mutex.go:87 +0x9d
sync.(*RWMutex).Lock(0xc42012e3c0)
  /usr/local/Cellar/go/1.8.1/libexec/src/sync/rwmutex.go:86 +0x2d
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).HasSynced(0xc42012e3c0, 0x1dcd400)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:167 +0x36
k8s.io/ingress/vendor/k8s.io/client-go/tools/cache.(*controller).HasSynced(0xc42020de80, 0xc420181001)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/client-go/tools/cache/controller.go:126 +0x33
k8s.io/ingress/core/pkg/ingress/controller.(*GenericController).controllersInSync(0xc4201e3180, 0x154c846)
  /Users/clucas/go/src/k8s.io/ingress/core/pkg/ingress/controller/controller.go:334 +0xc0
k8s.io/ingress/core/pkg/ingress/controller.(*GenericController).syncSecret(0xc4201e3180)
  /Users/clucas/go/src/k8s.io/ingress/core/pkg/ingress/controller/backend_ssl.go:40 +0x89
k8s.io/ingress/core/pkg/ingress/controller.(*GenericController).(k8s.io/ingress/core/pkg/ingress/controller.syncSecret)-fm()
  /Users/clucas/go/src/k8s.io/ingress/core/pkg/ingress/controller/controller.go:1202 +0x2a
k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420152f10)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x5e
k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420152f10, 0x2540be400, 0x0, 0x1, 0xc42004b860)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0xbd
k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc420152f10, 0x2540be400, 0xc42004b860)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d
k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait.Forever(0xc420152f10, 0x2540be400)
  /Users/clucas/go/src/k8s.io/ingress/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:43 +0x41
created by k8s.io/ingress/core/pkg/ingress/controller.GenericController.Start
  /Users/clucas/go/src/k8s.io/ingress/core/pkg/ingress/controller/controller.go:1202 +0x290

aledbf · 2017-05-29T16:15:37Z

@caseylucas please update the image to quay.io/aledbf/nginx-ingress-controller:0.130
I changed the sync logic in order to avoid calling the mentioned method every time we check the running configuration against the status in the server

caseylucas · 2017-05-29T16:21:56Z

i'll give it a try. It normally takes a while to see the problem. Can you send me a link to your changes (or a diff/patch) in case I want to merge them into my version that has a few more debug messages? Thanks! BTW.

aledbf · 2017-05-29T16:25:23Z

@caseylucas here #792

caseylucas · 2017-05-29T23:47:40Z

@aledbf I merged your changes into mine and It's been running for 7+ hours and still looking good. 😄 I'm surprised this hasn't bitten more people before - but good to get it fixed!

ese · 2017-05-30T18:11:06Z

I am still facing this using quay.io/aledbf/nginx-ingress-controller:0.130

caseylucas · 2017-05-30T19:49:41Z

@ese Can you get stack traces of an ingress that seems to be having the problem? I pulled one like this:
Find ingress pod and IP:

kubectl get pods | grep nginx
nginx-ingress-lb-2czvi                            1/1       Running   0          18h

kubectl describe pods nginx-ingress-lb-2czvi | grep IP
IP:		10.2.2.13

Dump goroutines:

curl http://10.2.2.13:10254/debug/pprof/goroutine?debug=2 > goroutines.txt

We should be able to see from the stack traces if you're seeing the same problem.

ese · 2017-05-30T21:01:36Z

@caseylucas Thanks for the tips to debugging. Here is the file from an nginx-controller with the problem
https://gist.github.com/ese/cbd8b31b2d215f6aed3f764e55af814b

caseylucas · 2017-05-31T02:56:06Z

@ese I verified that you are not seeing the exact same problem I was seeing. Sorry 😞 . The easy way to spot it is to find two DeltaFIFO in the same call stack.

caseylucas · 2017-05-31T02:59:28Z

@ese Can you confirm that you are seeing no more updates whatsoever to nginx.conf file once your ingress controller is messed up - even if you delete pods, make ingress definition changes, etc.?

caseylucas closed this as completed May 29, 2017

Platzii mentioned this issue May 30, 2017

[nginx] Pod IPs not being updated in upstreams #795

Closed

caseylucas mentioned this issue Jun 7, 2017

nginx-ingress: occasional 503 Service Temporarily Unavailable #821

Closed

caseylucas mentioned this issue Aug 15, 2017

Pods not routable after updating deployment #1135

Closed

alopsing mentioned this issue Jul 8, 2021

Ingress Controller forwarding traffic to a POD(IP) even after termination #7330

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upstream IP inconsistent with pod IP after pod deletion #768

upstream IP inconsistent with pod IP after pod deletion #768

caseylucas commented May 26, 2017

caseylucas commented May 28, 2017

aledbf commented May 28, 2017 •

edited

Loading

caseylucas commented May 28, 2017

caseylucas commented May 29, 2017

aledbf commented May 29, 2017

caseylucas commented May 29, 2017

aledbf commented May 29, 2017

caseylucas commented May 29, 2017

ese commented May 30, 2017

caseylucas commented May 30, 2017

ese commented May 30, 2017

caseylucas commented May 31, 2017

caseylucas commented May 31, 2017

upstream IP inconsistent with pod IP after pod deletion #768

upstream IP inconsistent with pod IP after pod deletion #768

Comments

caseylucas commented May 26, 2017

Problem

Log

Versions

Pod Info

nginx.conf oddness

caseylucas commented May 28, 2017

aledbf commented May 28, 2017 • edited Loading

caseylucas commented May 28, 2017

caseylucas commented May 29, 2017

aledbf commented May 29, 2017

caseylucas commented May 29, 2017

aledbf commented May 29, 2017

caseylucas commented May 29, 2017

ese commented May 30, 2017

caseylucas commented May 30, 2017

ese commented May 30, 2017

caseylucas commented May 31, 2017

caseylucas commented May 31, 2017

aledbf commented May 28, 2017 •

edited

Loading