Kong Ingress returns 500 error after 5 min

[javiramos1]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When we send a request we get a 500 HTTP error after exactly 5 minutes that is not coming from the app. Nor the service. It is coming from the Ingress

Expected Behavior

Respect the timeout annotations and do not timeout after 5 minutes.

Steps To Reproduce

We are using Kong in Kubernetes:


app.kubernetes.io/version: '3.2'
helm.sh/chart: kong-2.22.0

We are using AWS EKS and we have setup kong with service.beta.kubernetes.io/aws-load-balancer-type: nlb

We have an application with an Ingress and a Service.

The service has the timeout annotations:

konghq.com/read-timeout: '915000'
    konghq.com/send-timeout: '915000'

When we send a request we get a 500 HTTP error after exactly 5 minutes that is not coming from the app.
I tried to port-forward to the service and it works fine. It only happens when using the ingress.

If I set the timeout annotations to 3 minutes then I get 504 timeout error after 3 minutes as expected. If I set it to longer than 5 minutes, then I get a 500 error.

I tried to disable all plugins but it did not help. We have no idea why this is happening. Any help would be greatly appreciated since we are really stuck, thank you!

### Kong Ingress Controller version

```shell
3.2

kong-2.22.0

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.14", GitCommit:"3321ffc07d2f046afdf613796f9032f4460de093", GitTreeState:"clean", BuildDate:"2022-11-09T13:40:19Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"29+", GitVersion:"v1.29.4-eks-036c24b", GitCommit:"9c0e57823b31865d0ee095997d9e7e721ffdc77f", GitTreeState:"clean", BuildDate:"2024-04-30T23:53:58Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.29) exceeds the supported minor version skew of +/-1

Anything else?

No response

[randmonkey]

@javiramos1 Please confirm the version of your KIC and Kong gateway separately. The version 3.2 provided seems to be the version of Kong gateway.
You can use the following command to get the image used for Ingress controller and Kong gateway

kubectl get pods -n <ns> -l app.kubernetes.io/name=kong -o jsonpath={.spec.containers[*].image}

Also, konghq.com/send-timeout is not a supported annotation and Kong service does not have a send_timeout attribute. You can set konghq.com/connect-timeout or konghq.com/write-timeout: https://docs.konghq.com/kubernetes-ingress-controller/latest/reference/annotations/#konghqcomconnect-timeout. However, if your KIC version is below 2.8, they are not supported.

[javiramos1]

Hi, thanks for the reply. I will update the annotations and remove send-timeout.

We are using kong/kubernetes-ingress-controller:2.9 thanks!

[randmonkey]

@javiramos1 Can you please provide the further information for us to triage?

• The manifests of relevant Ingresses and Services
• The behavior of your backend services (Do they just receive requests and hang on the connection but writes nothing to the response?)
  BTW, the issue is also possible caused by AWS NLB's idle timeout on TCP connections. I found an article to tell about such example: https://medium.com/tenable-techblog/lessons-from-aws-nlb-timeouts-5028a8f65dda and the related document here https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout