-
Notifications
You must be signed in to change notification settings - Fork 593
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kong routes requests to targets deleted via Kong Ingress controller #6312
Comments
@zackery-parkhurst KIC does not watch for HPA events but watch for services and endpoints. After KIC noticed that a pod is deleted (or marked as not ready that reflected to a change on related |
@randmonkey Thank you for the response. I understand that KIC watches the endpoints to update the upstream targets. Its the 3 second sync period that is causing my issues. As when a pod is deleted or marked for deletion. It takes KIC ~3 seconds to then update Kong. Which means if any requests come in during that time. Kong will then send requests to a target that is being shut down. And if the request does not finish before the app shuts down its server. Then the request will just hang until the kong timeout is reached and kong kills it with a 504. Thank you for the information on We could shorten the So what would be the recommended way to handle this situation? As if there is no way to prevent kong from sending a request to a pod that is being shut down/terminated? Last thing: |
That is exactly what I was looking for. Thank you! |
Is there an existing issue for this?
Current Behavior
Whenever a scale down event happens on a kubernetes pod from HPA scaling down or a deployment deleting pods in update strategy kong sends request to the pod after its already been deleted. Resulting in a request that hangs for 60 seconds until eventually kong cancels and responds with 504.
Notice in these logs you can see that the pod is deleted. Kong Ingress Controller updates the configuration in kong. But a request comes through in between that time and just hangs for 60 seconds.
![image](https://private-user-images.githubusercontent.com/97055114/347465747-caea4414-2ed8-4ccd-9eab-2815cff10d2a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ1NzM2NTksIm5iZiI6MTcyNDU3MzM1OSwicGF0aCI6Ii85NzA1NTExNC8zNDc0NjU3NDctY2FlYTQ0MTQtMmVkOC00Y2NkLTllYWItMjgxNWNmZjEwZDJhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI1VDA4MDkxOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhZDg1MjczMTNkZjY2MmJiMWMxNmJiMDU3YjNiZmE5YjdiNzdhMGU4YjVjODI1NzkwMjcxMmI2MDAzNWE2NjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.iRpOM0vlqk2CeTygRADXXjGTi2Dach3M0bg10FOViu0)
Expected Behavior
Whenever I delete a pod, whether that happens through an HPA downscale event or a deployment. Kong should immediately no longer use the target ip.
However, there is a delay in the time from pod being deleted to kong updating its configuration. And if any requests come through during that time. They just hand and timeout.
Steps To Reproduce
Kong Ingress Controller version
Kubernetes version
Anything else?
The only other thing I would point out is that it appears kong does not sync the configuration until after the pod has been deleted. Which is odd as I though that kong watches the endpoint objects for changes. And the endpiont object is immediately updated the moment the pod is marked for deletion.
So theoretically kong should update itself before a pod is deleted. As a pod has a default termination grace period of 30 seconds. So when the pod is first marked for deletion the endpoint is immediately updated. Which should cause kong to start updating at that point.
And then on a final note. Regardless if kong is updating its configuration as soon as pod is marked for deletion. Or after its fully deleted still doesn't change the fact that there is always some latency between a pod being deleted or marked for deletion.....to kic updating its upstream targets and then sync'ing that with kong.
So technically there is always a window where a request can come in and hit a stale target and then sit and wait for 60 seconds(or however long timeout is for).
How can this be avoided? Or rectified as to not cause problems? As our issue is that kong takes 60 seconds waiting to timeout on an ip that no longer exists. The clients end up timing out or kong then times out and returns a 504.
The text was updated successfully, but these errors were encountered: